LFD Book Forum  

Go Back   LFD Book Forum > Course Discussions > Online LFD course > Homework 2

Reply
 
Thread Tools Display Modes
  #1  
Old 10-13-2013, 08:59 PM
yaser's Avatar
yaser yaser is offline
Caltech
 
Join Date: Aug 2009
Location: Pasadena, California, USA
Posts: 1,477
Default LR and PLA with scaled input space

A post at another forum:
Quote:
While answering question 7, I mistakenly took my test points from [0,1]\times[0,1], which resulted in the number of iterations being approximately double of what it was when I corrected it to [-1,1]\times[-1,1].

I thought that this might simply be because doubling the interval provided more area, making the points more loosely scattered and allowing for a larger number of lines to satisfy the test cases. However, when I tested my hypothesis by taking intervals of [-2,2]\times[-2,2] and [-4,4]\times[-4,4], etc., I found that the number of iterations was least for [-2,2]\times[-2,2] and went up in either direction from there. The points which defined my line were also taken from the larger intervals.

Can anyone help explain why this might be the case?
If you scale \bf x, then the linear regression solution \bf w scales in the opposite direction (other things being equal) since it is trying to make {\bf w}^{\rm T} {\bf  x} match the same value (+1 or -1). Now if you take the LR solution \bf w and use it as initial condition for PLA, the impact of each PLA iteration scales up with \bf x since you are adding (y {\bf x}) to the weight vector at each iteration.

Put these together and you conclude that, as \bf x scales up and down, the impact of the LR solution vector on PLA goes down and up, respectively, and significantly so. On the large \bf x extreme, the LR solution \bf w behaves like the vector \bf 0 so you get the original PLA iterations. As \bf x gets smaller, \bf w kicks in as a good initial condition (with non-trivial size) and you gain some PLA iterations. As \bf x diminishes, PLA will take longer to correct the misclassified points that the LR \bf w didn't get simply because the PLA iteration becomes relatively smaller in the movement that it creates.
__________________
Where everyone thinks alike, no one thinks very much
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT -7. The time now is 02:50 AM.


Powered by vBulletin® Version 3.8.3
Copyright ©2000 - 2019, Jelsoft Enterprises Ltd.
The contents of this forum are to be used ONLY by readers of the Learning From Data book by Yaser S. Abu-Mostafa, Malik Magdon-Ismail, and Hsuan-Tien Lin, and participants in the Learning From Data MOOC by Yaser S. Abu-Mostafa. No part of these contents is to be communicated or made accessible to ANY other person or entity.