View Single Post
Old 08-17-2012, 03:26 PM
yaser's Avatar
yaser yaser is offline
Join Date: Aug 2009
Location: Pasadena, California, USA
Posts: 1,478
Default Re: Overfitting with Polynomials : deterministic noise

Originally Posted by Kevin View Post
Chapter 4 state that:
- A 2nd order polynomial could be better than a 10th order polynomial to fit a 50th order polynomial target and it's due to the deterministic noise.

--> So I conclude that there is more deterministic noise with 10th order than with the 2nd order.
The confusion is justified, since there are two opposing factors here (see Exercise 4.3). There is more deterministic noise with the 2nd order model than with the 10th order model (which would suggest more overfitting with the 2nd order), but the model itself is simpler so that would suggest less overfitting. It turns out that the latter factor wins here.

If you want to isolate the impact of deterministic noise on overfitting without interference from the model complexity, you can fix the model and change the complexity of the target function.
Where everyone thinks alike, no one thinks very much
Reply With Quote