P14-17 Out-Of-Sample Data Set Size?

[munchkin]

Should the out-of-sample data set also be 100 randomly-generated points or should it be larger like in several of the earlier homeworks? Thanks for your attention.

[yaser]

Originally Posted by munchkin

Should the out-of-sample data set also be 100 randomly-generated points or should it be larger like in several of the earlier homeworks? Thanks for your attention.

Larger will give you a more reliable estimate of , and that may be necessary to make sure that the chosen answer is correct.

[JohnH]

The size of the out-of-sample set determines the accuracy and precision of the estimate of . I've generally used sets of at least 1000 points.

[Andrs]

In general the answers alternatives have good margins. If you run at least 1000 experiments, you should get "reasonable results" (E_out) on average with at least 100 test points. As always in machine learning, the more test data, the better.

[TonySuarez]

Originally Posted by munchkin

Should the out-of-sample data set also be 100 randomly-generated points or should it be larger like in several of the earlier homeworks? Thanks for your attention.

I settled in 200 points for testing, the same set for all batch of 1000 runs, and Eout seemed very stable.

[samirbajaj]

Originally Posted by Andrs

In general the answers alternatives have good margins. ...

For questions 17 and 18, my answers are different from one set of experiments to the next.

Can't say any more, but I'm wondering if anyone else had a similar experience.

Thanks.

-Samir