LFD Book Forum (http://book.caltech.edu/bookforum/index.php)
-   Chapter 5 - Three Learning Principles (http://book.caltech.edu/bookforum/forumdisplay.php?f=112)
-   -   Ecv versus Eout versus Etest (http://book.caltech.edu/bookforum/showthread.php?t=4457)

 ddas2 11-05-2013 01:10 PM

Ecv versus Eout versus Etest

I computed E_cv for part (d) in HW9. I compared the theoretical and estimated results and verfied they are same for all values of lambda. However, when I plot E_test(w_reg(lambda)) vs lambda against E_cv vs lambda, I observe E_test to be always larger than E_cv. I think it should not be the case because E_test is approximately E_out and E_out should be always less than E_cv.
I was wondering if I am making a mistake somewhere. I checked code etc. but cannot think of anyway to do some type of sanity check anywhere. Is there any way to know where the problem might be?

 magdon 11-06-2013 12:21 PM

Re: Ecv versus Eout versus Etest

Ecv is an estimate for Eout based on N-1 points. So you expect Eout<Ecv because Eout is based on learning from N points. However, Etest is just an estimate for Eout and can be higher or lower, though unbiased.

Also note that if there is data snooping, that is if the test set in [i]any[i] way affected the learning process that produces , then Etest can, and usually will, be affected.

Quote:
 Originally Posted by ddas2 (Post 11601) I computed E_cv for part (d) in HW9. I compared the theoretical and estimated results and verfied they are same for all values of lambda. However, when I plot E_test(w_reg(lambda)) vs lambda against E_cv vs lambda, I observe E_test to be always larger than E_cv. I think it should not be the case because E_test is approximately E_out and E_out should be always less than E_cv. I was wondering if I am making a mistake somewhere. I checked code etc. but cannot think of anyway to do some type of sanity check anywhere. Is there any way to know where the problem might be?

 alanericy 11-06-2013 07:54 PM

Re: Ecv versus Eout versus Etest

Hi Professor, for the 7th problem in HW9 the Ecv and Etest are classification error or regression error? Thanks!

 magdon 11-17-2013 10:55 AM

Re: Ecv versus Eout versus Etest

The digits problem is about classification, so all errors are classification errors.

However, one can us different algorithms to get your final classifier, one being the regression algorithm treating the as real values not binary classification values. The algorithm you use to get your final classifier is up to you, but the final result has to be a classifier.

Quote:
 Originally Posted by alanericy (Post 11603) Hi Professor, for the 7th problem in HW9 the Ecv and Etest are classification error or regression error? Thanks!

 All times are GMT -7. The time now is 11:10 AM.