optimistic bias of validation set error

[ilya239]

The lecture showed that error on the validation set underpredicts E_out, because the validation set was used to choose the best model. On the other hand, this error was obtained after training on only a subset of data, which would make the error over-predict E_out. Do these effects cancel out, or is the error determined from the validation set still an under-prediction?

[yaser]

Originally Posted by ilya239

The lecture showed that error on the validation set underpredicts E_out, because the validation set was used to choose the best model. On the other hand, this error was obtained after training on only a subset of data, which would make the error over-predict E_out. Do these effects cancel out, or is the error determined from the validation set still an under-prediction?

They do partially cancel out, but it is difficult to tell analytically which effect wins out.