Quote:
Originally Posted by jlaurentum
Hello All:
In minute 23:45 of Lecture 11, the restricted learner's reasoning is based on a rule of thumb, whereby you should have 10 data points for every parameter you want to estimate in your model. Where (in the book or in the other lectures) can I find more information on the justification for this rule of thumb?
|
Let me first point to that part of the lecture using the lecture tag:
The rule of thumb is a practical observation, so its real justification is simply that it has worked most of the time in practice. Once can justify the form that the number of examples is a multiple of the VC dimension by arguing that having multiple data points to fit per degree of freedom will force that degree of freedom to a 'compromise' that is likely to capture what is common between these data points, i.e., likely to generalize. Whether that multiple is 5 or 10 or 100, however, is an empirical observation that is difficult to reason about in a genuine way.