Libsvm

[itooam]

It is difficult to look at this homework and see how it applies to the real world. Being a non-US citizen, "symmetry" and "intensity", of zip codes means nothing? I assumed it probably meant nothing to anybody? So in reality given an unknown dataset and the question do a "0 versus all" you would probably apply weighting as you don't know anything about the data. I.e., consider a study where you are looking at heart-attack patients, it wouldn't be unreasonable to have 99% -1s and 1% 1s. This is a real issue for SVMs as these skewed classes will not focus in well on the 1s. So weighting is a good thing to use in the right circumstances.

Going back to the homework, the dataset probably has a similar distribution of 0s, 1s, 2s... 9s so in reality you probably wouldn't be asked by a client to "do 0 versus all", a real ask would be to "predict what classes each row belongs to". In which case the SVM should work well. But for the homework we are explicitly told to do this for binary which is fine enough - proof of concept and all that.

[Keith]

This data was introduced in lecture 3 as features of handwritten digits.

[itooam]

Ah, that's interesting. All my points still hold though.

I do agree though that for answering this homework "-wi" should not be used (as I wrote in my first post on this thread (#16)) as I'm sure it would have been specifically mentioned otherwise.

[itooam]

Apart from:
0 versus all
and
1 versus all

all the other "versus all" SVM class models for the training set are populated solely of -1s. This to me says either:
1) I am doing something wrong.
2) the skewed class problem I mentioned earlier is apparent.

Can anyone confirm they get the same results? I'd hate to fall at the first hurdle!

[htlin]

Originally Posted by itooam

Apart from:
0 versus all
and
1 versus all

all the other "versus all" SVM class models for the training set are populated solely of -1s. This to me says either:
1) I am doing something wrong.
2) the skewed class problem I mentioned earlier is apparent.

Can anyone confirm they get the same results? I'd hate to fall at the first hurdle!

The fact that the one-versus-all decomposition often results in unbalanced data sets is indeed often observed in practice, and causes trouble to most of the binary classification algorithms that focuses on error rates, not just SVM. There are many remedies to the trouble, such as viewing one-versus-all decomposition from the regression perspective rather than from the classification one.

Also, FYI, the following paper defends that one-versus-all can still be useful when applied with care.

http://jmlr.csail.mit.edu/papers/vol.../rifkin04a.pdf

Hope this helps.

[itooam]

Thanks very much for the link to that pdf, have been looking for something like that would describe the options in more detail... with pros and cons etc... Many thanks I think the math may beat me though, but maybe able to half understand.

I did see earlier another option in LIBSVM for the "svm_type":
2 -- one-class SVM ...from my intial research this sounds like it maybe similar to what you mentioned with regards to "regression perspective" instead of classification? This would certainly be another angle to look at instead of re-weighting data so that the classifiers are balanced (the -wi option I was going on about)... lots of further research...

[Andrs]

Originally Posted by jakvas

oh yeah you can balance using -wi but don't do it (if youd have to it would be stated in the homework) also there is no point in balancing since well.... any number occurs fewer times than all the others combined so in essence you'd be balancing a skew which should be there (also I think it might be some sort of data snooping(?))

anyway I found my problem hint: make sure your input has the correct form

If you display the data points, you will see that it is not only a question of there are too few +1 and a surplus of -1. The unbalance is really created because a lot of data points are very well mixed with no separation pattern at all in most of the cases (one exception). May be the poly with degree 2 and the C value is not suitable at all for this feature selection. In the case that you can see a separation of data, the svm is doing a reasonable job even if there is fewer +1.

[htlin]

Originally Posted by itooam

I did see earlier another option in LIBSVM for the "svm_type":
2 -- one-class SVM ...from my intial research this sounds like it maybe similar to what you mentioned with regards to "regression perspective" instead of classification? This would certainly be another angle to look at instead of re-weighting data so that the classifiers are balanced (the -wi option I was going on about)... lots of further research...

One-class SVM is for unsupervised learning (learning without labels).

For the regression perspective of one-versus-all, it may be worth referring to one of the earliest papers on one-versus-all, in which they use "regression decision tree" (called model tree) for one-versus-all.

@article{EF98,
title={{Using model trees for classification}},
author={Frank, E. and Wang, Y. and Inglis, S. and Holmes, G. and Witten, I.H.
},
journal={Machine Learning},
volume={32},
number={1},
pages={63--76},
year={1998},
publisher={Springer}
}

Hope this helps.

[itooam]

Thanks again, managed to find a copy here:
http://www.springerlink.com/content/...6/fulltext.pdf
My reading list is getting pretty long

[MLearning]

Originally Posted by invis

Pleasy, some one who understand how this libsvm works tell us how to go through week 8 problems.

I tryed to install libsvm in Octave, thats what I have:


Then using *.exe files:


actually dont know why the kernel is polynomial:
1 -- polynomial: (gamma*u'*v + coef0)^degree
but didnt find in list

Thats what I have in last iteration:

And features.model file didnt appears.

Actually I am prefer using my code from week7, but with upper bound C and some others. But this 7291 X 7291 matrix...

@invis,

If you look at the training data, it has no +1/-1 label. I think you think to transform the digits in to binary labels data. The classifier expected to see +1/-1 labels in the training data; hence, very low cross-validation accuracy when you apply SVM on the raw data.