LFD Book Forum I think Q9 average number of irritations is related to implementation of PLA?!

#1
07-12-2012, 04:21 PM
 hesam.creative Junior Member Join Date: Jul 2012 Posts: 7
I think Q9 average number of irritations is related to implementation of PLA?!

Hi every one!
I was wondering that the average number of irritations is kinda related to the way you exploit the PLA.Am I clear?
regards.
#2
07-12-2012, 06:50 PM
 yaser Caltech Join Date: Aug 2009 Location: Pasadena, California, USA Posts: 1,477
Re: I think Q9 average number of irritations is related to implementation of PLA?!

Quote:
 Originally Posted by hesam.creative Hi every one! I was wondering that the average number of irritations is kinda related to the way you exploit the PLA.Am I clear?
The homework specifies:

"Start the PLA with the weight vector being all zeros, and at each iteration have the algorithm choose a point randomly from the set of misclassi ed points."
__________________
Where everyone thinks alike, no one thinks very much
#3
07-13-2012, 08:50 AM
 hesam.creative Junior Member Join Date: Jul 2012 Posts: 7
Re: I think Q9 average number of irritations is related to implementation of PLA?!

Thanks professor,I think I have read the old version.I came up whit this idea:
Is it justified to compare Q7 through 10 to the probabilistic model in lecture #2,bin and marbles,say,all the points in [1,1]*[1,-1] as all marbles in bin,random points as random green and red marbles(a sample drawn from bin).p(f(x)~=g(x)) as μ. however,in this model f is determined,while target function is never known to us.Moreover,M is infinite;however,this is related to generalization theory.
#4
07-13-2012, 11:31 AM
 yaser Caltech Join Date: Aug 2009 Location: Pasadena, California, USA Posts: 1,477
Re: I think Q9 average number of irritations is related to implementation of PLA?!

Quote:
 Originally Posted by hesam.creative Is it justified to compare Q7 through 10 to the probabilistic model in lecture #2,bin and marbles,say,all the points in [1,1]*[1,-1] as all marbles in bin,random points as random green and red marbles(a sample drawn from bin).p(f(x)~=g(x)) as μ. however,in this model f is determined,while target function is never known to us.Moreover,M is infinite;however,this is related to generalization theory.
The theory will indeed address this question in detail in Lectures 5-7, and the perceptron case in particular will be analyzed in Lecture 7.
__________________
Where everyone thinks alike, no one thinks very much
#5
07-14-2012, 05:54 PM
 data_user Junior Member Join Date: Jul 2012 Posts: 6
Re: I think Q9 average number of irritations is related to implementation of PLA?!

Dear Yaser,

Thanks a lot for your very interesting and useful course.
I have a question on HW1,Q9.

I implemented PLA. The implementation looks correct: I answered Q7,Q8, and Q10 correctly. More importantly, I can see from a visualization (see below) that the PLA does its job. Here, dots are data, green line corresponds to the target function f, and 1000 yellow lines correspond to g_i, where i=1:1000.

In the first case, the average # of iterations is k1 (relatively small), in the second case it is k2 (relatively large), and k2 is approximately equal to 10*k1. Based on the first case, I had to chose one out 5 possible answers, while, based on the second case, I had to chose another. My (submitted) choice turned out to be wrong.

On the other hand, from the above figures, it is intuitively clear that to find g in the 2nd case is more difficult (yellow area is smaller), and, therefore, it should take more iterations for the PLA to converge. In general, isn't it true that the number of iterations significantly depends on the data? Say, it could be k but also it could be 10*k, depending on the sample x1,...,xN?

Am I missing something?
#6
07-14-2012, 08:28 PM
 yaser Caltech Join Date: Aug 2009 Location: Pasadena, California, USA Posts: 1,477
Re: I think Q9 average number of irritations is related to implementation of PLA?!

Quote:
 Originally Posted by data_user In general, isn't it true that the number of iterations significantly depends on the data? Say, it could be k but also it could be 10*k, depending on the sample x1,...,xN?
You are correct. The number of iterations does depend on the sample. The averaging over a large number of randomly generated samples should take care of these variations, though. Did you get two different answers after averaging over two sets of samples each generated according the the problem specs?

BTW, as alluded to in the preamble of the homework, the goal of the question is to make sure that you go through the experiment, analyze it, and understand the algorithm well. The plots you included in your post look great and they show that the goal was achieved. Having said that, the answer in the solution key should certainly be the correct answer.
__________________
Where everyone thinks alike, no one thinks very much
#7
07-15-2012, 04:08 AM
 data_user Junior Member Join Date: Jul 2012 Posts: 6
Re: I think Q9 average number of irritations is related to implementation of PLA?!

Quote:
 Originally Posted by yaser The averaging over a large number of randomly generated samples should take care of these variations, though. Did you get two different answers after averaging over two sets of samples each generated according the the problem specs?
Ah... I see my mistake now. I generated data x1,...,xn (uniformly on [-1,1]^2) and I fixed it forever. Then, I applied the PLA 1000 times to learn from the fixed data. (In every run, at each iteration, the PLA chooses the misclassified point randomly). In this way I obtain figure 1. Next, I generated new data x1',...,xn', fixed it, and applied the PLA 1000 times again. This resulted into figure 2.

Instead of fixing the sample, I should have generated new data for each PLA run!

Thank you! Looking forward to solving HW2

 Thread Tools Display Modes Linear Mode

 Posting Rules You may not post new threads You may not post replies You may not post attachments You may not edit your posts BB code is On Smilies are On [IMG] code is On HTML code is Off Forum Rules
 Forum Jump User Control Panel Private Messages Subscriptions Who's Online Search Forums Forums Home General     General Discussion of Machine Learning     Free Additional Material         Dynamic e-Chapters         Dynamic e-Appendices Course Discussions     Online LFD course         General comments on the course         Homework 1         Homework 2         Homework 3         Homework 4         Homework 5         Homework 6         Homework 7         Homework 8         The Final         Create New Homework Problems Book Feedback - Learning From Data     General comments on the book     Chapter 1 - The Learning Problem     Chapter 2 - Training versus Testing     Chapter 3 - The Linear Model     Chapter 4 - Overfitting     Chapter 5 - Three Learning Principles     e-Chapter 6 - Similarity Based Methods     e-Chapter 7 - Neural Networks     e-Chapter 8 - Support Vector Machines     e-Chapter 9 - Learning Aides     Appendix and Notation     e-Appendices

All times are GMT -7. The time now is 10:58 PM.