View Single Post
Old 11-06-2013, 01:21 PM
magdon's Avatar
magdon magdon is offline
Join Date: Aug 2009
Location: Troy, NY, USA.
Posts: 597
Default Re: Ecv versus Eout versus Etest

Ecv is an estimate for Eout based on N-1 points. So you expect Eout<Ecv because Eout is based on learning from N points. However, Etest is just an estimate for Eout and can be higher or lower, though unbiased.

Also note that if there is data snooping, that is if the test set in [i]any[i] way affected the learning process that produces g, then Etest can, and usually will, be affected.

Originally Posted by ddas2 View Post
I computed E_cv for part (d) in HW9. I compared the theoretical and estimated results and verfied they are same for all values of lambda. However, when I plot E_test(w_reg(lambda)) vs lambda against E_cv vs lambda, I observe E_test to be always larger than E_cv. I think it should not be the case because E_test is approximately E_out and E_out should be always less than E_cv.
I was wondering if I am making a mistake somewhere. I checked code etc. but cannot think of anyway to do some type of sanity check anywhere. Is there any way to know where the problem might be?
Have faith in probability
Reply With Quote