Quote:
Originally Posted by ilya239
That's like problem 1: if you try many enough different out-of-samples, inevitably there will be one on which your hypothesis does great, and one on which it does badly. As an extreme case, if your out-of-sample testing size was 1 instead of 1000, on some of these out-of-samples you'd get 0% error rate and on some you'd get 100% error rate. To get an actual estimate of out-of-sample error rate you should pool all these together.
|
This is not the case - I average 1000 out-of-samples anyway. Is seems the reason is large variation in in-sample error for different train samples.