Quote:
Originally Posted by yaser
Does variation in the average error mean that you repeat the entire experiment you described (including the target, training set, and resulting linear fit) and look at the different averages you get?

Not quite so. When I repeat entire experiment (including the target, training set, and resulting linear fit) I always get approximately same averages. I get different averages (0.01 ... 0.13) when I use one target and average over 1000 outofsample sets (for a single target).
Now as I plotted both lines (target and hypothesis) per your advice I begin to think that this is maybe what we should expect. Linear regression not always fits well. Usually it looks good giving small insample error. But sometimes disagreement is visually large (>0.1 insample error). This is the root of variation in average error when I use same "bad" regression for all 1000 iterations. I hope this type of experiment isn't implied by the problem. Otherwise it has no certain answer  at least 2 answers match.
So there is another question. Is >0.1 insample error and visually nonoptimal fit still valid outcome of linear regression for linearly separable data?