Validaten and VC-Bound in practice
 

Hello,

after reading the validation chapter of the book, I got some questions about the VC-bound.

I understand, that especially for cross-validation, the generalization theory is not completely applicable anymore. I think that's why your wrote the "model complexity penalty term" in all equations with the O(.) notation instead of using the exact bound from the second chapter (2.12).

However in a practical situation you must give an estimate about the E_out to the customer. And a constant factor like would make a direct difference in the numbers given to the customer. With O(.) notation you lose those factors, that leads to a more optimistic estimation.

So in practice, when using the validation error to estimate Eout for the customer, do you simply apply the full formula for the VC-Bound or Hoeffding-Bound (in case of discrete models to select)?

Or does the validation error just gives an relative estimate and when you have to provide absolute numbers to customer you have to use a separated test-set and apply the simple Hoeffding-Bound to Etest?

Best regards