Quote:
Originally Posted by jforbes
I used the same test set to evaluate the out of sample error for the alternatives, so it was in principle quite possible to get the same . I'm not sure if this was the appropriate approach.
In principle though, what you're doing with your test set is you're approximating the true value of , which you only approach in the limit of an infinite test set, in which case the probability that you get exactly the same should be vanishingly small unless your hypotheses were the same exact function, which won't happen in our situation.

I realise that, and was using a large test set, so the situation did not arise. What I did have on one question was a lot of unchanged
and different
. As I result, I was uncomfortable with my answer (although it was correct).