View Single Post
Old 05-10-2013, 08:56 AM
yaser's Avatar
yaser yaser is offline
Join Date: Aug 2009
Location: Pasadena, California, USA
Posts: 1,477
Default Re: Restricted Learner's Rule of Thumb (Lecture 11)

Originally Posted by jlaurentum View Post
Hello All:

In minute 23:45 of Lecture 11, the restricted learner's reasoning is based on a rule of thumb, whereby you should have 10 data points for every parameter you want to estimate in your model. Where (in the book or in the other lectures) can I find more information on the justification for this rule of thumb?
Let me first point to that part of the lecture using the lecture tag:

The rule of thumb is a practical observation, so its real justification is simply that it has worked most of the time in practice. Once can justify the form that the number of examples is a multiple of the VC dimension by arguing that having multiple data points to fit per degree of freedom will force that degree of freedom to a 'compromise' that is likely to capture what is common between these data points, i.e., likely to generalize. Whether that multiple is 5 or 10 or 100, however, is an empirical observation that is difficult to reason about in a genuine way.
Where everyone thinks alike, no one thinks very much
Reply With Quote