Quote:
Originally Posted by yaser
Just to clarify. You used the insample points to train and arrived at a final set of weights (corresponding to the final hypothesis). Each out ofsample point is now tested on this hypothesis and compared to the target value on the same point. Now, what exactly do you do to get the two scenarios you are describing?

1st (normal) scenario: I test outofsample data set (100 points) against linear model. I repeat it 1000 times: generate 100 insample points, linear fit, generate 100 outofsample points, test. On each iteration accumulate # of mistaken points. Average errors when done. Average error is stable from run to run.
2nd scenario: fit linear model only once. Repeat 1000 times: generate 100 outofsample points, test. Accumulate and average errors when done. Here I get remarkable variation in average error.
I'd like to understand why these scenarios different. I believe they must not