LFD Book Forum  

Go Back   LFD Book Forum > Course Discussions > Online LFD course > Homework 7

Reply
 
Thread Tools Display Modes
  #1  
Old 08-19-2012, 11:51 AM
rainbow rainbow is offline
Member
 
Join Date: Jul 2012
Posts: 41
Default SVM to return probabilistic output

Instead of using the SVM for pure classification, is it possible to return probabilities in the form

\frac{1}{1 + e^{-w^{'}x - b}}

or by any other transform?
Reply With Quote
  #2  
Old 08-19-2012, 03:45 PM
htlin's Avatar
htlin htlin is offline
NTU
 
Join Date: Aug 2009
Location: Taipei, Taiwan
Posts: 601
Default Re: SVM to return probabilistic output

Yes, the usual one used for SVMs is proposed by Platt:

http://citeseerx.ist.psu.edu/viewdoc...10.1.1.41.1639

which is of the form

\frac{1}{1 + \exp(- (A (\mathbf{w}^T \mathbf{x} + b) + B))}

and estimates A and B by a logistic-regression like optimization problem. An improved implementation for calculating A and B can be found in

Hsuan-Tien Lin, Chih-Jen Lin, and Ruby C. Weng. A Note on Platt's Probabilistic Outputs for Support Vector Machines. Machine Learning, 68(3), 267-276, 2007.

http://www.csie.ntu.edu.tw/~htlin/pa.../plattprob.pdf

Hope this helps.
__________________
When one teaches, two learn.
Reply With Quote
  #3  
Old 08-20-2012, 02:49 PM
rainbow rainbow is offline
Member
 
Join Date: Jul 2012
Posts: 41
Default Re: SVM to return probabilistic output

Thanks!
Reply With Quote
  #4  
Old 08-22-2012, 10:11 AM
patrickjtierney patrickjtierney is offline
Member
 
Join Date: Jul 2012
Location: Toronto, Canada
Posts: 33
Default Re: SVM to return probabilistic output

Yes. Thank you. Very interesting. I read both papers (well, skimmed some parts) and basically followed but I do have a general question.

I can understand A as a saturation factor or gain, but at first glance B is a little confusing. If B is non-zero, then the probability at the decision boundary will not be 1/2.

Is the reason for needing non-zero B that the mapping from Y->T no longer just maps +1 to 1, and -1 to 0, but rather to two values in (0,1) based on the relative number of +1s to -1s?
Reply With Quote
  #5  
Old 08-22-2012, 09:21 PM
samirbajaj samirbajaj is offline
Member
 
Join Date: Jul 2012
Location: Silicon Valley
Posts: 48
Default Re: SVM to return probabilistic output

And just out of curiosity - as an extension to the original question:

Can SVMs be used for regression? If so, do they perform better than the regression methods we have learned about in the course?

Thanks.

-Samir
Reply With Quote
  #6  
Old 08-23-2012, 03:25 AM
htlin's Avatar
htlin htlin is offline
NTU
 
Join Date: Aug 2009
Location: Taipei, Taiwan
Posts: 601
Default Re: SVM to return probabilistic output

Quote:
Originally Posted by patrickjtierney View Post
Yes. Thank you. Very interesting. I read both papers (well, skimmed some parts) and basically followed but I do have a general question.

I can understand A as a saturation factor or gain, but at first glance B is a little confusing. If B is non-zero, then the probability at the decision boundary will not be 1/2.

Is the reason for needing non-zero B that the mapping from Y->T no longer just maps +1 to 1, and -1 to 0, but rather to two values in (0,1) based on the relative number of +1s to -1s?
You are very right. My personal interpretation is that B provides an opportunity to calibrate the boundary of SVM for probability estimates. Recall that SVM roots from large-margin and hence the hyperplane is "right in the middle of the two classes." While arguably, for probability estimates, a good hyperplane (of P = \frac{1}{2}) shall be somewhat away from the majority class. So there may be a need to "shift" the hyperplane by B.

Hope this helps.
__________________
When one teaches, two learn.
Reply With Quote
  #7  
Old 08-23-2012, 03:28 AM
htlin's Avatar
htlin htlin is offline
NTU
 
Join Date: Aug 2009
Location: Taipei, Taiwan
Posts: 601
Default Re: SVM to return probabilistic output

Quote:
Originally Posted by samirbajaj View Post
And just out of curiosity - as an extension to the original question:

Can SVMs be used for regression? If so, do they perform better than the regression methods we have learned about in the course?

Thanks.

-Samir
Yes, there are several extensions of SVM for regression. One of which was proposed by the original SVM author, commonly named \epsilon-support vector regression. \epsilon-SVR can be found in common SVM packages such as LIBSVM and shares many interesting properties with the classification one. The other is extended from linear regression, commonly named least-square SVM.

Hope this helps.
__________________
When one teaches, two learn.
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT -7. The time now is 10:21 AM.


Powered by vBulletin® Version 3.8.3
Copyright ©2000 - 2019, Jelsoft Enterprises Ltd.
The contents of this forum are to be used ONLY by readers of the Learning From Data book by Yaser S. Abu-Mostafa, Malik Magdon-Ismail, and Hsuan-Tien Lin, and participants in the Learning From Data MOOC by Yaser S. Abu-Mostafa. No part of these contents is to be communicated or made accessible to ANY other person or entity.