LFD Book Forum

LFD Book Forum (http://book.caltech.edu/bookforum/index.php)
-   Homework 7 (http://book.caltech.edu/bookforum/forumdisplay.php?f=136)
-   -   optimistic bias of validation set error (http://book.caltech.edu/bookforum/showthread.php?t=4009)

ilya239 02-19-2013 03:56 PM
optimistic bias of validation set error
 
The lecture showed that error on the validation set underpredicts E_out, because the validation set was used to choose the best model. On the other hand, this error was obtained after training on only a subset of data, which would make the error over-predict E_out. Do these effects cancel out, or is the error determined from the validation set still an under-prediction?

yaser 02-19-2013 04:31 PM
Re: optimistic bias of validation set error
 
Quote:
Originally Posted by ilya239 (Post 9441)
The lecture showed that error on the validation set underpredicts E_out, because the validation set was used to choose the best model. On the other hand, this error was obtained after training on only a subset of data, which would make the error over-predict E_out. Do these effects cancel out, or is the error determined from the validation set still an under-prediction?
They do partially cancel out, but it is difficult to tell analytically which effect wins out.


All times are GMT -7. The time now is 01:44 PM.

Powered by vBulletin® Version 3.8.3
Copyright ©2000 - 2019, Jelsoft Enterprises Ltd.
The contents of this forum are to be used ONLY by readers of the Learning From Data book by Yaser S. Abu-Mostafa, Malik Magdon-Ismail, and Hsuan-Tien Lin, and participants in the Learning From Data MOOC by Yaser S. Abu-Mostafa. No part of these contents is to be communicated or made accessible to ANY other person or entity.