
#1




Ecv versus Eout versus Etest
I computed E_cv for part (d) in HW9. I compared the theoretical and estimated results and verfied they are same for all values of lambda. However, when I plot E_test(w_reg(lambda)) vs lambda against E_cv vs lambda, I observe E_test to be always larger than E_cv. I think it should not be the case because E_test is approximately E_out and E_out should be always less than E_cv.
I was wondering if I am making a mistake somewhere. I checked code etc. but cannot think of anyway to do some type of sanity check anywhere. Is there any way to know where the problem might be? 
#2




Re: Ecv versus Eout versus Etest
Ecv is an estimate for Eout based on N1 points. So you expect Eout<Ecv because Eout is based on learning from N points. However, Etest is just an estimate for Eout and can be higher or lower, though unbiased.
Also note that if there is data snooping, that is if the test set in [i]any[i] way affected the learning process that produces , then Etest can, and usually will, be affected. Quote:
__________________
Have faith in probability 
#3




Re: Ecv versus Eout versus Etest
Hi Professor, for the 7th problem in HW9 the Ecv and Etest are classification error or regression error? Thanks!

#4




Re: Ecv versus Eout versus Etest
The digits problem is about classification, so all errors are classification errors.
However, one can us different algorithms to get your final classifier, one being the regression algorithm treating the as real values not binary classification values. The algorithm you use to get your final classifier is up to you, but the final result has to be a classifier.
__________________
Have faith in probability 
Thread Tools  
Display Modes  

