LFD Book Forum doubt in lecture 11, deterministic noise
 Register FAQ Calendar Mark Forums Read

#1
05-11-2012, 03:38 PM
 rohanag Invited Guest Join Date: Apr 2012 Posts: 94
doubt in lecture 11, deterministic noise

In the lecture 11, it is mentioned that deterministic noise depends on the hypothesis set, and deterministic noise decreases as the hypothesis set becomes more complex, because the set can tackle some of the deterministic noise.
But the experiment performed on slide 7 of lecture 11 suggests that the more complex model fails badly when compared to the less complex model.

The above statements seem to contradict in my mind, I cannot put them together. Can anyone please clear my doubt. Thank you.
#2
05-11-2012, 03:58 PM
 AqibEjaz Junior Member Join Date: May 2012 Posts: 7
Re: doubt in lecture 11, deterministic noise

This is indeed confusing and after spending some time thinking about this point I think I have finally understood it (I hope). Deterministic noise is nothing but the bias in the modelling hypothesis. So the more complex model will indeed have less deterministic noise (small bias). But this does not imply that this model will also have smaller Eout. Because Eout also depends on the variance of the hypothesis and since the variance of the more complex model will be large for small N, this means that Eout will be large for more complex model. But if we have sufficiently large sample size (i.e. large N) then both the variance and the bias (i.e. deterministic noise) will be small for complex model. Hence in this case the more complex model will outperform the simpler model. So the lesson learnt is: Complex model is better than simple model provided we have sufficient data. For small data sets, complex models overfit and it is better to choose simple models.
#3
05-11-2012, 06:25 PM
 yaser Caltech Join Date: Aug 2009 Location: Pasadena, California, USA Posts: 1,478
Re: doubt in lecture 11, deterministic noise

Quote:
 Originally Posted by rohanag In the lecture 11, it is mentioned that deterministic noise depends on the hypothesis set, and deterministic noise decreases as the hypothesis set becomes more complex
There are two factors here: the level of deterministic noise, and the complexity of the hypothesis set used to fit the data. Both of these factors are affected by having the target fixed while expanding the hypothesis set. The two effects push overfitting in different directions.

If you want to see the impact of deterministic noise by itself, without having another factor that affects overfitting in play, you should fix the hypothesis set and increase the complexity of the target function.
__________________
Where everyone thinks alike, no one thinks very much
#4
05-12-2012, 03:44 AM
 rohanag Invited Guest Join Date: Apr 2012 Posts: 94
Re: doubt in lecture 11, deterministic noise

Quote:
 Originally Posted by yaser If you want to see the impact of deterministic noise by itself, without having another factor that affects overfitting in play, you should fix the hypothesis set and increase the complexity of the target function.
got it, thanks Professor. So if we fix the H set and increase complexity of the target function, out overfit measure will increase.

Can we say the overfit measure, indicates noise?
#5
05-12-2012, 03:53 AM
 yaser Caltech Join Date: Aug 2009 Location: Pasadena, California, USA Posts: 1,478
Re: doubt in lecture 11, deterministic noise

Quote:
 Originally Posted by rohanag got it, thanks Professor. So if we fix the H set and increase complexity of the target function, out overfit measure will increase. Can we say the overfit measure, indicates noise?
It does. The specific way it indicates it is still a research topic.
__________________
Where everyone thinks alike, no one thinks very much
#6
05-12-2012, 03:56 AM
 rohanag Invited Guest Join Date: Apr 2012 Posts: 94
Re: doubt in lecture 11, deterministic noise

Quote:
 Originally Posted by yaser There are two factors here: the level of deterministic noise, and the complexity of the hypothesis set used to fit the data. Both of these factors are affected by having the target fixed while expanding the hypothesis set. The two effects push overfitting in different directions.
Thanks again Professor, I have just one more question.

The two effects push overfitting in different directions, this means that if we fix a target, and expand H, the more complex H has a better ability to tackle the deterministic noise, but the complexity of the H in turn, makes for bad overfitting?
#7
05-12-2012, 04:03 AM
 yaser Caltech Join Date: Aug 2009 Location: Pasadena, California, USA Posts: 1,478
Re: doubt in lecture 11, deterministic noise

Quote:
 Originally Posted by rohanag Thanks again Professor, I have just one more question. The two effects push overfitting in different directions, this means that if we fix a target, and expand H, the more complex H has a better ability to tackle the deterministic noise, but the complexity of the H in turn, makes for bad overfitting?
The second part is accurate as you phrased it. The first part I would rephrase by saying that the more complex is dealing with lower deterministic noise.
__________________
Where everyone thinks alike, no one thinks very much
#8
05-12-2012, 03:28 PM
 mic00 Invited Guest Join Date: Apr 2012 Posts: 49
Re: doubt in lecture 11, deterministic noise

Is there anything interesting to be said about the relationship between overfitting and the difference between and the truly best hypothesis in ? I think I've seen some others say that the deterministic noise is not dependent on the size of the data set, but I am wondering whether this difference is what accounts for the overfitting caused by deterministic noise. E.g. for complex enough target function, if our sample data set is very big, and we have a simple model (high bias/deterministic noise), we are not really overfitting, as I understand it.

Edit: I've been assuming that the definitions of bias and variance take expectations over all data sets of a particular fixed size -- I don't think this was explicitly stated, but I also don't think it makes sense otherwise. In homework #4, I computed a value for that was very far from the best possible hypothesis (lowest mean squared error), because was so low in that case.
#9
08-20-2012, 10:08 PM
 menon.rx Junior Member Join Date: Jul 2012 Posts: 4
Re: doubt in lecture 11, deterministic noise

I think I am still a bit unclear about deterministic noise. Doesn't the amount of noise (deterministic or stochastic) depend on both the bias and variance of the noise? For a given N, the more complex noise will have a higher variance but lower bias. Hence doesn't the amount of noise depend upon N and the relative complexity difference between the target function and the hypothesis (the level of deterministic noise if you will)?
#10
08-21-2012, 02:12 AM
 Andrs Member Join Date: Jul 2012 Posts: 47
Re: doubt in lecture 11, deterministic noise

In general a more complex H implies lower "deterministic noise" but it is important to take into consideration the amount of training data that you have (N) when discussing Eout. In the example shown in lecture 11 the target function was very complex (50th order) and the training data was noiseless. We could see that a simple hypothesis (second order pol) gave a much better Eout than the more complex hypothesis (10th order polynomial). In this case there was only "deterministic noise" and the more complex Hypothesis performed much worse even if the "deterministic noise" was lower for the more complex H.

 Tags deterministic noise, lecture 11

 Thread Tools Display Modes Linear Mode

 Posting Rules You may not post new threads You may not post replies You may not post attachments You may not edit your posts BB code is On Smilies are On [IMG] code is On HTML code is Off Forum Rules
 Forum Jump User Control Panel Private Messages Subscriptions Who's Online Search Forums Forums Home General     General Discussion of Machine Learning     Free Additional Material         Dynamic e-Chapters         Dynamic e-Appendices Course Discussions     Online LFD course         General comments on the course         Homework 1         Homework 2         Homework 3         Homework 4         Homework 5         Homework 6         Homework 7         Homework 8         The Final         Create New Homework Problems Book Feedback - Learning From Data     General comments on the book     Chapter 1 - The Learning Problem     Chapter 2 - Training versus Testing     Chapter 3 - The Linear Model     Chapter 4 - Overfitting     Chapter 5 - Three Learning Principles     e-Chapter 6 - Similarity Based Methods     e-Chapter 7 - Neural Networks     e-Chapter 8 - Support Vector Machines     e-Chapter 9 - Learning Aides     Appendix and Notation     e-Appendices

All times are GMT -7. The time now is 04:33 PM.