#11




Re: HW 2 Problem 6
Quote:
2nd scenario: fit linear model only once. Repeat 1000 times: generate 100 outofsample points, test. Accumulate and average errors when done. Here I get remarkable variation in average error. I'd like to understand why these scenarios different. I believe they must not 
#12




Re: HW 2 Problem 6
Does variation in the average error mean that you repeat the entire experiment you described (including the target, training set, and resulting linear fit) and look at the different averages you get?
__________________
Where everyone thinks alike, no one thinks very much 
#13




Re: HW 2 Problem 6
Quote:
Now as I plotted both lines (target and hypothesis) per your advice I begin to think that this is maybe what we should expect. Linear regression not always fits well. Usually it looks good giving small insample error. But sometimes disagreement is visually large (>0.1 insample error). This is the root of variation in average error when I use same "bad" regression for all 1000 iterations. I hope this type of experiment isn't implied by the problem. Otherwise it has no certain answer  at least 2 answers match. So there is another question. Is >0.1 insample error and visually nonoptimal fit still valid outcome of linear regression for linearly separable data? 
#14




Re: HW 2 Problem 6
Quote:

#15




Re: HW 2 Problem 6
Quote:

#16




Re: HW 2 Problem 6
I also observe some discrepancy while computing Eout. When I hold the target function fixed, Eout is approximately equal to Ein. When I use different target functions for each experiment, Eout is significantly higher than Ein. Is this expected?

#17




Re: HW 2 Problem 6
@MLearning
In my opinion E_out should be near E_in when target is not fixed(i.e averaging over 1000 iterations) . Unfortunately problem can be anywhere , but most probably in computing error. Could it be that when you are computing error for E_out for one iteration, the number of sample points for out of sample are 1000 , and if you forgot to change the number of samples points from 100 (from Q 5) to 1000 , then may be that is the cause of difference. (misclassified/sample_size) I am just guessing , since I make this kind of mistakes often. 
#18




Re: HW 2 Problem 6
@dsvav,
Thank you for your comments. You were right, I did forget to change the sample number (N) to 1000. But that doesn't change the result. It is possible that Eout is not the same as Ein although that is what we want. Indeed, we are applying linear regression to a random data that it hasn't seen before; hence, the larger deviation between Eout and Ein. 
#19




Re: HW 2 Problem 6
@MLearning
When I compute the difference between E_in and E_out I get the difference to be around 0.01. I still think difference should not be significant , does not this comes from Hoeffding Inequality ? Also since we are suppressing the "very bad event happening" and "very good event happening" by taking average over 1000 runs , so E_out should track E_in. This is my understanding , there is good chance that I am wrong By the way what is the difference you are getting ? 
#20




Re: HW 2 Problem 6
For a given target and regression Ein and Eout must not deviate much from each other for large N. The intuition is error zone between two lines is fixed, and points in Ein and Eout distributed uniformly.

Thread Tools  
Display Modes  

