LFD Book Forum  

Go Back   LFD Book Forum > Course Discussions > Online LFD course > Homework 2

Reply
 
Thread Tools Display Modes
  #1  
Old 10-15-2012, 08:11 PM
gah44 gah44 is offline
Invited Guest
 
Join Date: Jul 2012
Location: Seattle, WA
Posts: 153
Default Regression then PLA

In the Homework 1 section there is discussion showing that PLA is
independent of alpha, as only the ratio to w0 counts.

It seems, though, when using regression before PLA, as
initial weights for the PLA, that alpha is important again.

Too big, and the initial weights don't do much, too small
and it converges slow.
Reply With Quote
  #2  
Old 10-15-2012, 08:39 PM
yaser's Avatar
yaser yaser is offline
Caltech
 
Join Date: Aug 2009
Location: Pasadena, California, USA
Posts: 1,477
Default Re: Regression then PLA

Quote:
Originally Posted by gah44 View Post
In the Homework 1 section there is discussion showing that PLA is
independent of alpha, as only the ratio to w0 counts.

It seems, though, when using regression before PLA, as
initial weights for the PLA, that alpha is important again.

Too big, and the initial weights don't do much, too small
and it converges slow.
Good point.
__________________
Where everyone thinks alike, no one thinks very much
Reply With Quote
  #3  
Old 01-17-2013, 10:08 AM
Anne Paulson Anne Paulson is offline
Senior Member
 
Join Date: Jan 2013
Location: Silicon Valley
Posts: 52
Default Re: Regression then PLA

I'm coming up against this as well. Maybe I have a bug, but I'm finding that even though the regression finds an almost perfect line with, usually, very few points misclassified, I give the weights from the regression to PLA as initial weights and the PLA line bounces all over the place before settling down.

Scaling the regression weights up by a factor of 10 or 100 would speed up the PLA a lot, I think, by preventing the PLA update from moving the weights so much. That would have a similar effect to using a small alpha. But we're not supposed to do either thing, right?
Reply With Quote
  #4  
Old 01-17-2013, 10:27 AM
yaser's Avatar
yaser yaser is offline
Caltech
 
Join Date: Aug 2009
Location: Pasadena, California, USA
Posts: 1,477
Default Re: Regression then PLA

Quote:
Originally Posted by Anne Paulson View Post
I'm coming up against this as well. Maybe I have a bug, but I'm finding that even though the regression finds an almost perfect line with, usually, very few points misclassified, I give the weights from the regression to PLA as initial weights and the PLA line bounces all over the place before settling down.

Scaling the regression weights up by a factor of 10 or 100 would speed up the PLA a lot, I think, by preventing the PLA update from moving the weights so much. That would have a similar effect to using a small alpha. But we're not supposed to do either thing, right?
You are right, there is no scaling in Problem 7. Here, and in all homework problems, you are encouraged to explore outside the statement of the problem, like you have done here, but the choice of answer should follow the problem statement.
__________________
Where everyone thinks alike, no one thinks very much
Reply With Quote
  #5  
Old 01-20-2013, 02:47 AM
gah44 gah44 is offline
Invited Guest
 
Join Date: Jul 2012
Location: Seattle, WA
Posts: 153
Default Re: Regression then PLA

Seems to me that what it does is either give 0 iterations (if none are misclassified) or about as many as it did without the regression solution.

So what we are really counting is how often it gives 0 iterations, but yes, we have to follow the problem statement.
Reply With Quote
  #6  
Old 04-10-2013, 01:17 PM
Michael Reach Michael Reach is offline
Senior Member
 
Join Date: Apr 2013
Location: Baltimore, Maryland, USA
Posts: 71
Default Re: Regression then PLA

Huh - I haven't done this problem yet, but this is really interesting. I had thought the answer to this question would be, like, 1, but now I see that the first step is likely to ruin the advantage that the linear regression gave you. The PLA isn't initially very subtle; the weights eventually get big and you have to wait till the size of the adjustments becomes small relative to the weights for the fine tuning.
Reply With Quote
  #7  
Old 04-10-2013, 01:26 PM
Rahul Sinha Rahul Sinha is offline
Junior Member
 
Join Date: Apr 2013
Posts: 9
Default Re: Regression then PLA

Quote:
Originally Posted by Michael Reach View Post
Huh - I haven't done this problem yet, but this is really interesting. I had thought the answer to this question would be, like, 1, but now I see that the first step is likely to ruin the advantage that the linear regression gave you. The PLA isn't initially very subtle; the weights eventually get big and you have to wait till the size of the adjustments becomes small relative to the weights for the fine tuning.

For 10 data points, it might very well converge quickly! My simulation converges at the speed of light!
1000 data points is another story.
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT -7. The time now is 11:14 AM.


Powered by vBulletin® Version 3.8.3
Copyright ©2000 - 2019, Jelsoft Enterprises Ltd.
The contents of this forum are to be used ONLY by readers of the Learning From Data book by Yaser S. Abu-Mostafa, Malik Magdon-Ismail, and Hsuan-Tien Lin, and participants in the Learning From Data MOOC by Yaser S. Abu-Mostafa. No part of these contents is to be communicated or made accessible to ANY other person or entity.