View Single Post
Old 02-19-2013, 04:31 PM
yaser's Avatar
yaser yaser is offline
Join Date: Aug 2009
Location: Pasadena, California, USA
Posts: 1,477
Default Re: optimistic bias of validation set error

Originally Posted by ilya239 View Post
The lecture showed that error on the validation set underpredicts E_out, because the validation set was used to choose the best model. On the other hand, this error was obtained after training on only a subset of data, which would make the error over-predict E_out. Do these effects cancel out, or is the error determined from the validation set still an under-prediction?
They do partially cancel out, but it is difficult to tell analytically which effect wins out.
Where everyone thinks alike, no one thinks very much
Reply With Quote