View Single Post
  #2  
Old 11-06-2013, 01:21 PM
magdon's Avatar
magdon magdon is offline
RPI
 
Join Date: Aug 2009
Location: Troy, NY, USA.
Posts: 595
Default Re: Ecv versus Eout versus Etest

Ecv is an estimate for Eout based on N-1 points. So you expect Eout<Ecv because Eout is based on learning from N points. However, Etest is just an estimate for Eout and can be higher or lower, though unbiased.

Also note that if there is data snooping, that is if the test set in [i]any[i] way affected the learning process that produces g, then Etest can, and usually will, be affected.

Quote:
Originally Posted by ddas2 View Post
I computed E_cv for part (d) in HW9. I compared the theoretical and estimated results and verfied they are same for all values of lambda. However, when I plot E_test(w_reg(lambda)) vs lambda against E_cv vs lambda, I observe E_test to be always larger than E_cv. I think it should not be the case because E_test is approximately E_out and E_out should be always less than E_cv.
I was wondering if I am making a mistake somewhere. I checked code etc. but cannot think of anyway to do some type of sanity check anywhere. Is there any way to know where the problem might be?
__________________
Have faith in probability
Reply With Quote