Why not use soft margin SVM everytime?
Hello all,
I wanted to get your thoughts on somethings the professor said at the end of lecture 15. He said that if the data set is not linearly separable, we can still try the hard margin SVM (notwithstanding the fact that the quadratic programming package would probably complain in that case), and just check the results. I am assuming that by that he meant that E_in would turn out to be rather dismal in that case. He also said later that the value of C in soft margin SVM can be had from cross validation. My question comes from a long held feeling of general discomfort I have had about nonlinear transformation (NLT). If somebody decides to go to NLT, then she/he must have done so by data snooping, otherwise how would she know that a linear model wouldn't fit? The idea of using validation to decide the number of terms in a possible NLT seems to offer some respite here, by indicating, for example, that the linear model suffices (lowest cross validation error on the linear model). In the same vein, could we "always" use soft margin SVMs and use cross validation for getting C? That way, if the data set is in fact linearly separable, the result for C would hopefully be 0, reducing the problem to hardmargin? Thanks for your time. 
Re: Why not use soft margin SVM everytime?
The remark about using hardmargin SVM in nonseparable cases was not a recommendation. It was to point out that if you use hardmargin SVM and the data turns out to be nonseparable after all, this can be easily detected by checking the solution (if QP returns one) on the training data points. This relieves you from the need to determine linear separability before you apply hardmargin SVM. You can of course use softmargin SVM in all cases.

Re: Why not use soft margin SVM everytime?
Thank you very much, Professor. You are right, the QP solution would probably not return anything reasonable.
As I mentioned earlier, we don't know about the linear separability of the data, or at least, we can't know without looking, which would amount to snooping. And it is in those cases that I feel that the technique of cross validation is invaluable, as it can help one choose among the different kinds of models. In one of your earlier lectures, you had indicated that the linear models work surprisingly well in most real cases, and we even had a linear model (logistic regression) to handle noise. Do SVMs work well in noise, too? If they do, I wonder why anyone would use the traditional linear models when one can use the power of SVMs. 
Re: Why not use soft margin SVM everytime?
Quote:
In general if I have something that needs soft outputs I'll use logistic regression, and in other cases I prefer SVM. But the preference is more personal than objective. :) From the perspective of optimization, logistic regression and linear regression are arguably easier problems than linear SVM, by the way. But nowadays such a difference of difficulty in optimization is usually not a big deal. Nonlinear SVM is another story. With the power of kernels, the overfitting problem needs to be resolved more carefully by parameter/kernel selection, and the optimization problem becomes much harder to solve. Those are part of the reasons that the linear family (including the linear SVM) can and shall still be a firsthand choice. Hope this helps. 
Re: Why not use soft margin SVM everytime?
Thank you very much for your input, Prof. Lin.

All times are GMT 7. The time now is 01:46 PM. 
Powered by vBulletin® Version 3.8.3
Copyright ©2000  2020, Jelsoft Enterprises Ltd.
The contents of this forum are to be used ONLY by readers of the Learning From Data book by Yaser S. AbuMostafa, Malik MagdonIsmail, and HsuanTien Lin, and participants in the Learning From Data MOOC by Yaser S. AbuMostafa. No part of these contents is to be communicated or made accessible to ANY other person or entity.