Thread: HW 2 Problem 6
View Single Post
Old 07-24-2012, 01:02 AM
rakhlin rakhlin is offline
Join Date: Jun 2012
Posts: 24
Default Re: HW 2 Problem 6

Originally Posted by ilya239 View Post
That's like problem 1: if you try many enough different out-of-samples, inevitably there will be one on which your hypothesis does great, and one on which it does badly. As an extreme case, if your out-of-sample testing size was 1 instead of 1000, on some of these out-of-samples you'd get 0% error rate and on some you'd get 100% error rate. To get an actual estimate of out-of-sample error rate you should pool all these together.
This is not the case - I average 1000 out-of-samples anyway. Is seems the reason is large variation in in-sample error for different train samples.
Reply With Quote