LFD Book Forum

LFD Book Forum (http://book.caltech.edu/bookforum/index.php)
-   Homework 7 (http://book.caltech.edu/bookforum/forumdisplay.php?f=136)
-   -   Odd results using LibSVM on Prob 8 (http://book.caltech.edu/bookforum/showthread.php?t=4018)

apinde 02-22-2013 10:38 AM

Odd results using LibSVM on Prob 8
 
I generated 10 real number pairs between [-1,+1] and labelled them using the function f = 1+1.91*x1 + 2.13*x2. Used LibSVM package in a Windows C++ environment setting SVM_Type as C_SVC and kernel_type as LINEAR. Called svm_train() to train on this data. It returned 6 SVs and model parameters. Using the same data set for testing (called svm_test()), I found that 2 out of the 10 points are misclassified! Happens quite frequently on repeating this process by randomly changing the function F() and the data. That's quite a poor result given that a PLA can easily get to 100% classification on the training sets. I have to believe I'm doing something wrong. Has anyone used LibSVM in this way? Would appreciate any help, tips or pointers. This is driving me up the wall! Thanks.

yaser 02-22-2013 11:44 AM

Re: Odd results using LibSVM on Prob 8
 
Quote:

Originally Posted by apinde (Post 9468)
Using the same data set for testing (called svm_test()), I found that 2 out of the 10 points are misclassified!

In this homework, where we are using "hard-margin" SVM (the type covered in Lecture 14, as opposed to "soft-margin" which is covered next week) and with the data set being linearly separable by design, the in-sample error has to be zero. There has to be a bug somewhere. Hopefully others can share their experience with other QP packages as well.

ilya239 02-22-2013 04:17 PM

Re: Odd results using LibSVM on Prob 8
 
I'm using a different package (python and cvxopt), but had the same issue (some points would occasionally be misclassified). Turned out to be a bug in my code, in the way I was computing b once the \alpha have been returned from the quadratic programming call.

Try checking whether the returned solution satisfies the constraints, and whether it is at least a local minimum (small changes in \alpha lead to small increases in \cal L(\alpha). Also, that all but a few of the \alpha_i are near-zero. If all that is true, the error is downstream from there, else upstream.

jain.anand@tcs.com 02-22-2013 09:19 PM

Re: Odd results using LibSVM on Prob 8
 
I am using svm command in R using e1071 package. I get good results from it. But I found that for "linear" kernel I need to set the cost (i.e. C) value very high (e.g. 200 to 2000) to get a good fit and low number of support vectors. The problem did not specify we should be using high C for model fitting. Is that a practical observation for a linear kernel?

ilya239 02-22-2013 09:27 PM

Re: Odd results using LibSVM on Prob 8
 
Quote:

Originally Posted by jain.anand@tcs.com (Post 9488)
I am using svm command in R using e1071 package. I get good results from it. But I found that for "linear" kernel I need to set the cost (i.e. C) value very high (e.g. 200 to 2000) to get a good fit and low number of support vectors. The problem did not specify we should be using high C for model fitting. Is that a practical observation for a linear kernel?

Maybe C is the parameter for "soft margin" SVMs from next lecture, and setting it high restricts solutions to "hard margin" SVMs used in this problem?

Anne Paulson 02-22-2013 10:16 PM

Re: Odd results using LibSVM on Prob 8
 
I'm getting pretty poor results from qp() in Octave: so far it has classified my points correctly, but frequently the line is in the wrong place, so that it is not equally far from the support points. I found from discussion by the previous class that this is a known problem with the Octave implementation, and there is a trick (which I haven't tried yet) to make Octave perform better.

So it wouldn't surprise me if other languages also were giving so-so results, and that this is a problem that has nothing to do with student bugs. On the other hand, lots of people have bugs in their code...

apinde 02-23-2013 08:52 AM

Re: Odd results using LibSVM on Prob 8
 
Thanks for your suggestions, Ilya, Anand & Anne.

I'll try making the cost parameter C high. So far I only tried C=1.0 and 0.5
I've seen that the SV's that the LibSVM package identifies cannot be equidistant to any linear equation. All their alpha values are identical.

In my implementation I do the training, which creates the model and then do a test with the exact same file as the training file. The testing uses the model file created by LibSVM. Thus alpha's and the parameter b etc. are transparent and I don't mess with them. The only possibility of errors is if some wrong parameter or errors in the input.

colinpriest 02-24-2013 01:28 PM

Re: Odd results using LibSVM on Prob 8
 
Thanks for the hint on the cost parameter - that finally got my R code to produce sensible results :-)

apinde 02-25-2013 06:46 AM

Re: Odd results using LibSVM on Prob 8
 
Just wanted to close the loop. The cost parameter C was the culprit! When it is > ~50.0 then the results make sense and the number of SVs and alpha values stabilizes. Around C=15, the the number of SV's begins to increase and results get poorer. I think the comment that high C forces a hard margin solution is possibly the explanation. I noticed that at lower values of C the number of iterations used in getting a solution is also lower.

Thanks all.


All times are GMT -7. The time now is 07:49 AM.

Powered by vBulletin® Version 3.8.3
Copyright ©2000 - 2020, Jelsoft Enterprises Ltd.
The contents of this forum are to be used ONLY by readers of the Learning From Data book by Yaser S. Abu-Mostafa, Malik Magdon-Ismail, and Hsuan-Tien Lin, and participants in the Learning From Data MOOC by Yaser S. Abu-Mostafa. No part of these contents is to be communicated or made accessible to ANY other person or entity.