Ecv versus Eout versus Etest

[ddas2]

I computed E_cv for part (d) in HW9. I compared the theoretical and estimated results and verfied they are same for all values of lambda. However, when I plot E_test(w_reg(lambda)) vs lambda against E_cv vs lambda, I observe E_test to be always larger than E_cv. I think it should not be the case because E_test is approximately E_out and E_out should be always less than E_cv.
I was wondering if I am making a mistake somewhere. I checked code etc. but cannot think of anyway to do some type of sanity check anywhere. Is there any way to know where the problem might be?

[magdon]

Ecv is an estimate for Eout based on N-1 points. So you expect Eout<Ecv because Eout is based on learning from N points. However, Etest is just an estimate for Eout and can be higher or lower, though unbiased.

Also note that if there is data snooping, that is if the test set in [i]any[i] way affected the learning process that produces , then Etest can, and usually will, be affected.

Originally Posted by ddas2

I computed E_cv for part (d) in HW9. I compared the theoretical and estimated results and verfied they are same for all values of lambda. However, when I plot E_test(w_reg(lambda)) vs lambda against E_cv vs lambda, I observe E_test to be always larger than E_cv. I think it should not be the case because E_test is approximately E_out and E_out should be always less than E_cv.
I was wondering if I am making a mistake somewhere. I checked code etc. but cannot think of anyway to do some type of sanity check anywhere. Is there any way to know where the problem might be?

[alanericy]

Hi Professor, for the 7th problem in HW9 the Ecv and Etest are classification error or regression error? Thanks!

[magdon]

The digits problem is about classification, so all errors are classification errors.

However, one can us different algorithms to get your final classifier, one being the regression algorithm treating the as real values not binary classification values. The algorithm you use to get your final classifier is up to you, but the final result has to be a classifier.

Originally Posted by alanericy

Hi Professor, for the 7th problem in HW9 the Ecv and Etest are classification error or regression error? Thanks!