LFD Book Forum - View Single Post

Elroch · #3 06-11-2013, 11:59 AM

Quote:

Originally Posted by jforbes

I used the same test set to evaluate the out of sample error for the alternatives, so it was in principle quite possible to get the same $E_{out}$ . I'm not sure if this was the appropriate approach.

In principle though, what you're doing with your test set is you're approximating the true value of $E_{out}$ , which you only approach in the limit of an infinite test set, in which case the probability that you get exactly the same $E_{out}$ should be vanishingly small unless your hypotheses were the same exact function, which won't happen in our situation.

I realise that, and was using a large test set, so the situation did not arise. What I did have on one question was a lot of unchanged $E_{in}$ and different $E_{out}$ . As I result, I was uncomfortable with my answer (although it was correct).