Take the 2-minute tour ×
Quantitative Finance Stack Exchange is a question and answer site for finance professionals and academics. It's 100% free, no registration required.

I want to compare different error rates of different classifiers with the error rate from a weak learner (better than random guessing). So, my question is, what are a few choices for a simple, easy to process weak learner? Or, do I understand the concept incorrectly, and is a weak learner simply any benchmark that I choose (for example, a linear regression)?

If this question belongs to a different stackexchange, please let me know!

share|improve this question
1  
I believe this question is better suited for Stack Overflow. –  Good Guy Mike Dec 6 '13 at 22:05
1  
stats.stackexchange.com –  user2763361 Dec 9 '13 at 8:08
add comment

1 Answer

up vote 1 down vote accepted

A classifier can be weak for a number of reasons, and it mainly depends on characteristics of the data. For example, if the data are not linearly separable, then linear regression will be weak (poor correlation between predicted class and true class labels). However, if the data are linearly separable, then other classifiers may not work as well as linear regression. If you used ensemble methods (committee of classifiers), you could compare results across classifiers. You didn't mention anything about boosting, which is fundamental to weak learner applications. Without going into detail, you could start with the most basic classifiers such as k-nearest neighbors(kNN), Naive Bayes classifier(NBC), learning vector quantization(LVQ), linear discriminant analysis(LDA), and then linear regression. If the data depart from normality, then LDA may breakdown since covariance matrices are used, and if nonlinearly-separable data are present, regression may have larger error. Certainly, as you increase classifier complexity such as with support vector machines, random forests, artificial neural networks, the earlier mentioned classifiers (kNN, NBC, LDA, LREG) may not work as well. The main issue is that, as soon as you mention "weakness," the theory of boosting overarches everything you are doing, and boosting involves more complex methods for making a weak learner stronger. Hence, to pursue what you want to do, you may get "trapped" in boosting theory and become forced to only consider boosting issues -- which has its own unique assumptions. Make sure you really have to work with a weak classifier instead of comparing classifiers via an ensemble. (See Kuncheva's papers on the diversity issue for an ensemble of classifiers).

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.