LFD Book Forum  

Go Back   LFD Book Forum > Course Discussions > Online LFD course > Homework 6

Reply
 
Thread Tools Display Modes
  #1  
Old 05-25-2013, 11:08 AM
a.sanyal902 a.sanyal902 is offline
Member
 
Join Date: Apr 2013
Posts: 11
Default Doubt in Lecture 11

There was a previous thread (here) which discussed this, but I still had a nagging doubt.
Say the complexity of our hypothesis set matches that of the target function (or the set includes the target function). So, there is no deterministic noise. Moreover, let us assume there is no stochastic noise either.

However, due to a finite data set, we may still not be able to generalize very well. Is this still called overfitting? We referred to overfitting when the algorthm tries to select a hypothesis which fits the "noise", stochastic or deterministic. But there is no noise in the example above. We may call it variance, because we have many possible choices and few data points, but are we "overfitting" ?
Reply With Quote
  #2  
Old 05-25-2013, 03:57 PM
Elroch Elroch is offline
Invited Guest
 
Join Date: Mar 2013
Posts: 143
Default Re: Doubt in Lecture 11

Quote:
Originally Posted by a.sanyal902 View Post
There was a previous thread (here) which discussed this, but I still had a nagging doubt.
Say the complexity of our hypothesis set matches that of the target function (or the set includes the target function). So, there is no deterministic noise. Moreover, let us assume there is no stochastic noise either.

However, due to a finite data set, we may still not be able to generalize very well. Is this still called overfitting? We referred to overfitting when the algorthm tries to select a hypothesis which fits the "noise", stochastic or deterministic. But there is no noise in the example above. We may call it variance, because we have many possible choices and few data points, but are we "overfitting" ?
Firstly, you can have deterministic noise even if the exact target function is in a hypothesis set. The definition is based on the difference between an average hypothesis and the target function. This average hypothesis is not definable in terms of the hypothesis set: it requires, a set D of samples, a probability distribution \Psi on D and an algorithm M which associates a hypothesis with each element of D. Then it is defined as the function each of whose values is the average over D with respect to \Psi of the hypotheses M generates.

Even if there is no deterministic noise, this certainly doesn't preclude the possibility of overfitting: this merely means by comparison with some other machine M', M gives lower in sample error, but greater out of sample error.
Reply With Quote
  #3  
Old 05-25-2013, 10:39 PM
yaser's Avatar
yaser yaser is offline
Caltech
 
Join Date: Aug 2009
Location: Pasadena, California, USA
Posts: 1,474
Default Re: Doubt in Lecture 11

Quote:
Originally Posted by a.sanyal902 View Post
Say the complexity of our hypothesis set matches that of the target function (or the set includes the target function). So, there is no deterministic noise. Moreover, let us assume there is no stochastic noise either.

However, due to a finite data set, we may still not be able to generalize very well. Is this still called overfitting? We referred to overfitting when the algorthm tries to select a hypothesis which fits the "noise", stochastic or deterministic. But there is no noise in the example above. We may call it variance, because we have many possible choices and few data points, but are we "overfitting" ?
Since the model is fixed (and assuming no "early stopping" within this model), it will be just bad generalization, rather than overfitting. Recall that overfitting means getting worse E_{\rm out} as you get better E_{\rm in}, not just getting bad E_{\rm out} in the absolute.
__________________
Where everyone thinks alike, no one thinks very much
Reply With Quote
Reply

Tags
lecture 11

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT -7. The time now is 09:14 PM.


Powered by vBulletin® Version 3.8.3
Copyright ©2000 - 2018, Jelsoft Enterprises Ltd.
The contents of this forum are to be used ONLY by readers of the Learning From Data book by Yaser S. Abu-Mostafa, Malik Magdon-Ismail, and Hsuan-Tien Lin, and participants in the Learning From Data MOOC by Yaser S. Abu-Mostafa. No part of these contents is to be communicated or made accessible to ANY other person or entity.