LFD Book Forum Doubt in Lecture 11
 User Name Remember Me? Password
 FAQ Calendar Mark Forums Read

 Thread Tools Display Modes
#1
05-25-2013, 12:08 PM
 a.sanyal902 Member Join Date: Apr 2013 Posts: 11
Doubt in Lecture 11

There was a previous thread (here) which discussed this, but I still had a nagging doubt.
Say the complexity of our hypothesis set matches that of the target function (or the set includes the target function). So, there is no deterministic noise. Moreover, let us assume there is no stochastic noise either.

However, due to a finite data set, we may still not be able to generalize very well. Is this still called overfitting? We referred to overfitting when the algorthm tries to select a hypothesis which fits the "noise", stochastic or deterministic. But there is no noise in the example above. We may call it variance, because we have many possible choices and few data points, but are we "overfitting" ?
#2
05-25-2013, 04:57 PM
 Elroch Invited Guest Join Date: Mar 2013 Posts: 143
Re: Doubt in Lecture 11

Quote:
 Originally Posted by a.sanyal902 There was a previous thread (here) which discussed this, but I still had a nagging doubt. Say the complexity of our hypothesis set matches that of the target function (or the set includes the target function). So, there is no deterministic noise. Moreover, let us assume there is no stochastic noise either. However, due to a finite data set, we may still not be able to generalize very well. Is this still called overfitting? We referred to overfitting when the algorthm tries to select a hypothesis which fits the "noise", stochastic or deterministic. But there is no noise in the example above. We may call it variance, because we have many possible choices and few data points, but are we "overfitting" ?
Firstly, you can have deterministic noise even if the exact target function is in a hypothesis set. The definition is based on the difference between an average hypothesis and the target function. This average hypothesis is not definable in terms of the hypothesis set: it requires, a set of samples, a probability distribution on and an algorithm which associates a hypothesis with each element of . Then it is defined as the function each of whose values is the average over with respect to of the hypotheses generates.

Even if there is no deterministic noise, this certainly doesn't preclude the possibility of overfitting: this merely means by comparison with some other machine , gives lower in sample error, but greater out of sample error.
#3
05-25-2013, 11:39 PM
 yaser Caltech Join Date: Aug 2009 Location: Pasadena, California, USA Posts: 1,477
Re: Doubt in Lecture 11

Quote:
 Originally Posted by a.sanyal902 Say the complexity of our hypothesis set matches that of the target function (or the set includes the target function). So, there is no deterministic noise. Moreover, let us assume there is no stochastic noise either. However, due to a finite data set, we may still not be able to generalize very well. Is this still called overfitting? We referred to overfitting when the algorthm tries to select a hypothesis which fits the "noise", stochastic or deterministic. But there is no noise in the example above. We may call it variance, because we have many possible choices and few data points, but are we "overfitting" ?
Since the model is fixed (and assuming no "early stopping" within this model), it will be just bad generalization, rather than overfitting. Recall that overfitting means getting worse as you get better , not just getting bad in the absolute.
__________________
Where everyone thinks alike, no one thinks very much

 Tags lecture 11

 Thread Tools Display Modes Linear Mode

 Posting Rules You may not post new threads You may not post replies You may not post attachments You may not edit your posts BB code is On Smilies are On [IMG] code is On HTML code is Off Forum Rules
 Forum Jump User Control Panel Private Messages Subscriptions Who's Online Search Forums Forums Home General     General Discussion of Machine Learning     Free Additional Material         Dynamic e-Chapters         Dynamic e-Appendices Course Discussions     Online LFD course         General comments on the course         Homework 1         Homework 2         Homework 3         Homework 4         Homework 5         Homework 6         Homework 7         Homework 8         The Final         Create New Homework Problems Book Feedback - Learning From Data     General comments on the book     Chapter 1 - The Learning Problem     Chapter 2 - Training versus Testing     Chapter 3 - The Linear Model     Chapter 4 - Overfitting     Chapter 5 - Three Learning Principles     e-Chapter 6 - Similarity Based Methods     e-Chapter 7 - Neural Networks     e-Chapter 8 - Support Vector Machines     e-Chapter 9 - Learning Aides     Appendix and Notation     e-Appendices

All times are GMT -7. The time now is 03:52 AM.

 Contact Us - LFD Book - Top