LFD Book Forum Why not use soft margin SVM everytime?

#1
02-22-2013, 05:53 AM
 kartikeya_t@yahoo.com Member Join Date: Jul 2012 Posts: 17
Why not use soft margin SVM everytime?

Hello all,
I wanted to get your thoughts on somethings the professor said at the end of lecture 15. He said that if the data set is not linearly separable, we can still try the hard margin SVM (notwithstanding the fact that the quadratic programming package would probably complain in that case), and just check the results. I am assuming that by that he meant that E_in would turn out to be rather dismal in that case.
He also said later that the value of C in soft margin SVM can be had from cross validation.
My question comes from a long held feeling of general discomfort I have had about non-linear transformation (NLT). If somebody decides to go to NLT, then she/he must have done so by data snooping, otherwise how would she know that a linear model wouldn't fit? The idea of using validation to decide the number of terms in a possible NLT seems to offer some respite here, by indicating, for example, that the linear model suffices (lowest cross validation error on the linear model).
In the same vein, could we "always" use soft margin SVMs and use cross validation for getting C? That way, if the data set is in fact linearly separable, the result for C would hopefully be 0, reducing the problem to hard-margin?
#2
02-22-2013, 10:50 AM
 yaser Caltech Join Date: Aug 2009 Location: Pasadena, California, USA Posts: 1,477
Re: Why not use soft margin SVM everytime?

The remark about using hard-margin SVM in non-separable cases was not a recommendation. It was to point out that if you use hard-margin SVM and the data turns out to be non-separable after all, this can be easily detected by checking the solution (if QP returns one) on the training data points. This relieves you from the need to determine linear separability before you apply hard-margin SVM. You can of course use soft-margin SVM in all cases.
__________________
Where everyone thinks alike, no one thinks very much
#3
02-22-2013, 11:25 AM
 kartikeya_t@yahoo.com Member Join Date: Jul 2012 Posts: 17
Re: Why not use soft margin SVM everytime?

Thank you very much, Professor. You are right, the QP solution would probably not return anything reasonable.
As I mentioned earlier, we don't know about the linear separability of the data, or at least, we can't know without looking, which would amount to snooping.
And it is in those cases that I feel that the technique of cross validation is invaluable, as it can help one choose among the different kinds of models.
In one of your earlier lectures, you had indicated that the linear models work surprisingly well in most real cases, and we even had a linear model (logistic regression) to handle noise. Do SVMs work well in noise, too? If they do, I wonder why anyone would use the traditional linear models when one can use the power of SVMs.
#4
02-22-2013, 04:38 PM
 htlin NTU Join Date: Aug 2009 Location: Taipei, Taiwan Posts: 601
Re: Why not use soft margin SVM everytime?

Quote:
 Originally Posted by kartikeya_t@yahoo.com Thank you very much, Professor. You are right, the QP solution would probably not return anything reasonable. As I mentioned earlier, we don't know about the linear separability of the data, or at least, we can't know without looking, which would amount to snooping. And it is in those cases that I feel that the technique of cross validation is invaluable, as it can help one choose among the different kinds of models. In one of your earlier lectures, you had indicated that the linear models work surprisingly well in most real cases, and we even had a linear model (logistic regression) to handle noise. Do SVMs work well in noise, too? If they do, I wonder why anyone would use the traditional linear models when one can use the power of SVMs.
Linear (soft-margin) SVM is indeed a competitive model in the linear family. In my own experience, linear SVM and logistic regression are comparable to each other in practice. In fact if you check the error function similar to what we do in Exercise 3.9 of the LFD book, you'll see the similarity between SVM and (regularized) logistic regression.

In general if I have something that needs soft outputs I'll use logistic regression, and in other cases I prefer SVM. But the preference is more personal than objective. :-)

From the perspective of optimization, logistic regression and linear regression are arguably easier problems than linear SVM, by the way. But nowadays such a difference of difficulty in optimization is usually not a big deal.

Nonlinear SVM is another story. With the power of kernels, the overfitting problem needs to be resolved more carefully by parameter/kernel selection, and the optimization problem becomes much harder to solve. Those are part of the reasons that the linear family (including the linear SVM) can and shall still be a first-hand choice.

Hope this helps.
__________________
When one teaches, two learn.
#5
02-22-2013, 04:43 PM
 htlin NTU Join Date: Aug 2009 Location: Taipei, Taiwan Posts: 601
Re: Why not use soft margin SVM everytime?

Quote:
 Originally Posted by kartikeya_t@yahoo.com . In the same vein, could we "always" use soft margin SVMs and use cross validation for getting C? That way, if the data set is in fact linearly separable, the result for C would hopefully be 0, reducing the problem to hard-margin? Thanks for your time.
Just to clarify, if the data is linearly separable, the corresponding that would work would be any greater than or equal to (of the hard-margin solution), not .

In practice it would be difficult to find true hard-margin SVM solvers, and indeed soft-margin works better than hard-margin most of the time (or in some sense soft-margin includes hard-margin as a "special case" when the data is separable). So the procedure that you describe is indeed what most people do for using SVMs.

Hope this helps.
__________________
When one teaches, two learn.
#6
02-22-2013, 05:36 PM
 kartikeya_t@yahoo.com Member Join Date: Jul 2012 Posts: 17
Re: Why not use soft margin SVM everytime?

Thank you very much for your input, Prof. Lin.

 Thread Tools Display Modes Linear Mode

 Posting Rules You may not post new threads You may not post replies You may not post attachments You may not edit your posts BB code is On Smilies are On [IMG] code is On HTML code is Off Forum Rules
 Forum Jump User Control Panel Private Messages Subscriptions Who's Online Search Forums Forums Home General     General Discussion of Machine Learning     Free Additional Material         Dynamic e-Chapters         Dynamic e-Appendices Course Discussions     Online LFD course         General comments on the course         Homework 1         Homework 2         Homework 3         Homework 4         Homework 5         Homework 6         Homework 7         Homework 8         The Final         Create New Homework Problems Book Feedback - Learning From Data     General comments on the book     Chapter 1 - The Learning Problem     Chapter 2 - Training versus Testing     Chapter 3 - The Linear Model     Chapter 4 - Overfitting     Chapter 5 - Three Learning Principles     e-Chapter 6 - Similarity Based Methods     e-Chapter 7 - Neural Networks     e-Chapter 8 - Support Vector Machines     e-Chapter 9 - Learning Aides     Appendix and Notation     e-Appendices

All times are GMT -7. The time now is 05:59 AM.