LFD Book Forum  

Go Back   LFD Book Forum > Course Discussions > Online LFD course > Homework 8

Reply
 
Thread Tools Display Modes
  #21  
Old 08-31-2012, 11:27 AM
itooam itooam is offline
Senior Member
 
Join Date: Jul 2012
Posts: 100
Default Re: Libsvm

It is difficult to look at this homework and see how it applies to the real world. Being a non-US citizen, "symmetry" and "intensity", of zip codes means nothing? I assumed it probably meant nothing to anybody? So in reality given an unknown dataset and the question do a "0 versus all" you would probably apply weighting as you don't know anything about the data. I.e., consider a study where you are looking at heart-attack patients, it wouldn't be unreasonable to have 99% -1s and 1% 1s. This is a real issue for SVMs as these skewed classes will not focus in well on the 1s. So weighting is a good thing to use in the right circumstances.

Going back to the homework, the dataset probably has a similar distribution of 0s, 1s, 2s... 9s so in reality you probably wouldn't be asked by a client to "do 0 versus all", a real ask would be to "predict what classes each row belongs to". In which case the SVM should work well. But for the homework we are explicitly told to do this for binary which is fine enough - proof of concept and all that.
Reply With Quote
  #22  
Old 08-31-2012, 11:33 AM
Keith Keith is offline
Member
 
Join Date: Jul 2012
Posts: 16
Default Re: Libsvm

This data was introduced in lecture 3 as features of handwritten digits.
Reply With Quote
  #23  
Old 08-31-2012, 11:40 AM
itooam itooam is offline
Senior Member
 
Join Date: Jul 2012
Posts: 100
Default Re: Libsvm

Ah, that's interesting. All my points still hold though.

I do agree though that for answering this homework "-wi" should not be used (as I wrote in my first post on this thread (#16)) as I'm sure it would have been specifically mentioned otherwise.
Reply With Quote
  #24  
Old 08-31-2012, 02:59 PM
itooam itooam is offline
Senior Member
 
Join Date: Jul 2012
Posts: 100
Default Re: Libsvm

Apart from:
0 versus all
and
1 versus all

all the other "versus all" SVM class models for the training set are populated solely of -1s. This to me says either:
1) I am doing something wrong.
2) the skewed class problem I mentioned earlier is apparent.

Can anyone confirm they get the same results? I'd hate to fall at the first hurdle!
Reply With Quote
  #25  
Old 09-01-2012, 08:07 AM
htlin's Avatar
htlin htlin is offline
NTU
 
Join Date: Aug 2009
Location: Taipei, Taiwan
Posts: 601
Default Re: Libsvm

Quote:
Originally Posted by itooam View Post
Apart from:
0 versus all
and
1 versus all

all the other "versus all" SVM class models for the training set are populated solely of -1s. This to me says either:
1) I am doing something wrong.
2) the skewed class problem I mentioned earlier is apparent.

Can anyone confirm they get the same results? I'd hate to fall at the first hurdle!
The fact that the one-versus-all decomposition often results in unbalanced data sets is indeed often observed in practice, and causes trouble to most of the binary classification algorithms that focuses on error rates, not just SVM. There are many remedies to the trouble, such as viewing one-versus-all decomposition from the regression perspective rather than from the classification one.

Also, FYI, the following paper defends that one-versus-all can still be useful when applied with care.

http://jmlr.csail.mit.edu/papers/vol.../rifkin04a.pdf

Hope this helps.
__________________
When one teaches, two learn.
Reply With Quote
  #26  
Old 09-01-2012, 10:28 AM
itooam itooam is offline
Senior Member
 
Join Date: Jul 2012
Posts: 100
Default Re: Libsvm

Thanks very much for the link to that pdf, have been looking for something like that would describe the options in more detail... with pros and cons etc... Many thanks I think the math may beat me though, but maybe able to half understand.

I did see earlier another option in LIBSVM for the "svm_type":
2 -- one-class SVM ...from my intial research this sounds like it maybe similar to what you mentioned with regards to "regression perspective" instead of classification? This would certainly be another angle to look at instead of re-weighting data so that the classifiers are balanced (the -wi option I was going on about)... lots of further research...
Reply With Quote
  #27  
Old 09-01-2012, 02:07 PM
Andrs Andrs is offline
Member
 
Join Date: Jul 2012
Posts: 47
Default Re: Libsvm

Quote:
Originally Posted by jakvas View Post
oh yeah you can balance using -wi but don't do it (if youd have to it would be stated in the homework) also there is no point in balancing since well.... any number occurs fewer times than all the others combined so in essence you'd be balancing a skew which should be there (also I think it might be some sort of data snooping(?))

anyway I found my problem hint: make sure your input has the correct form
If you display the data points, you will see that it is not only a question of there are too few +1 and a surplus of -1. The unbalance is really created because a lot of data points are very well mixed with no separation pattern at all in most of the cases (one exception). May be the poly with degree 2 and the C value is not suitable at all for this feature selection. In the case that you can see a separation of data, the svm is doing a reasonable job even if there is fewer +1.
Reply With Quote
  #28  
Old 09-02-2012, 05:31 AM
htlin's Avatar
htlin htlin is offline
NTU
 
Join Date: Aug 2009
Location: Taipei, Taiwan
Posts: 601
Default Re: Libsvm

Quote:
Originally Posted by itooam View Post
I did see earlier another option in LIBSVM for the "svm_type":
2 -- one-class SVM ...from my intial research this sounds like it maybe similar to what you mentioned with regards to "regression perspective" instead of classification? This would certainly be another angle to look at instead of re-weighting data so that the classifiers are balanced (the -wi option I was going on about)... lots of further research...
One-class SVM is for unsupervised learning (learning without labels).

For the regression perspective of one-versus-all, it may be worth referring to one of the earliest papers on one-versus-all, in which they use "regression decision tree" (called model tree) for one-versus-all.

@article{EF98,
title={{Using model trees for classification}},
author={Frank, E. and Wang, Y. and Inglis, S. and Holmes, G. and Witten, I.H.
},
journal={Machine Learning},
volume={32},
number={1},
pages={63--76},
year={1998},
publisher={Springer}
}

Hope this helps.
__________________
When one teaches, two learn.
Reply With Quote
  #29  
Old 09-03-2012, 03:44 AM
itooam itooam is offline
Senior Member
 
Join Date: Jul 2012
Posts: 100
Default Re: Libsvm

Thanks again, managed to find a copy here:
http://www.springerlink.com/content/...6/fulltext.pdf
My reading list is getting pretty long
Reply With Quote
  #30  
Old 09-03-2012, 02:16 PM
MLearning MLearning is offline
Senior Member
 
Join Date: Jul 2012
Posts: 56
Default Re: Libsvm

Quote:
Originally Posted by invis View Post
Pleasy, some one who understand how this libsvm works tell us how to go through week 8 problems.

I tryed to install libsvm in Octave, thats what I have:


Then using *.exe files:


actually dont know why the kernel is polynomial:
1 -- polynomial: (gamma*u'*v + coef0)^degree
but didnt find (1+ x_n^T x_m)^Q in list

Thats what I have in last iteration:

And features.model file didnt appears.

Actually I am prefer using my code from week7, but with upper bound C and some others. But this 7291 X 7291 matrix...
@invis,

If you look at the training data, it has no +1/-1 label. I think you think to transform the digits in to binary labels data. The classifier expected to see +1/-1 labels in the training data; hence, very low cross-validation accuracy when you apply SVM on the raw data.
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT -7. The time now is 10:35 AM.


Powered by vBulletin® Version 3.8.3
Copyright ©2000 - 2020, Jelsoft Enterprises Ltd.
The contents of this forum are to be used ONLY by readers of the Learning From Data book by Yaser S. Abu-Mostafa, Malik Magdon-Ismail, and Hsuan-Tien Lin, and participants in the Learning From Data MOOC by Yaser S. Abu-Mostafa. No part of these contents is to be communicated or made accessible to ANY other person or entity.