LFD Book Forum

LFD Book Forum (http://book.caltech.edu/bookforum/index.php)
-   Homework 6 (http://book.caltech.edu/bookforum/forumdisplay.php?f=135)
-   -   doubt in lecture 11, deterministic noise (http://book.caltech.edu/bookforum/showthread.php?t=479)

rohanag 05-11-2012 02:38 PM

doubt in lecture 11, deterministic noise
 
In the lecture 11, it is mentioned that deterministic noise depends on the hypothesis set, and deterministic noise decreases as the hypothesis set becomes more complex, because the set can tackle some of the deterministic noise.
But the experiment performed on slide 7 of lecture 11 suggests that the more complex model fails badly when compared to the less complex model.

The above statements seem to contradict in my mind, I cannot put them together. Can anyone please clear my doubt. Thank you.

AqibEjaz 05-11-2012 02:58 PM

Re: doubt in lecture 11, deterministic noise
 
This is indeed confusing and after spending some time thinking about this point I think I have finally understood it (I hope). Deterministic noise is nothing but the bias in the modelling hypothesis. So the more complex model will indeed have less deterministic noise (small bias). But this does not imply that this model will also have smaller Eout. Because Eout also depends on the variance of the hypothesis and since the variance of the more complex model will be large for small N, this means that Eout will be large for more complex model. But if we have sufficiently large sample size (i.e. large N) then both the variance and the bias (i.e. deterministic noise) will be small for complex model. Hence in this case the more complex model will outperform the simpler model. So the lesson learnt is: Complex model is better than simple model provided we have sufficient data. For small data sets, complex models overfit and it is better to choose simple models.

yaser 05-11-2012 05:25 PM

Re: doubt in lecture 11, deterministic noise
 
Quote:

Originally Posted by rohanag (Post 2057)
In the lecture 11, it is mentioned that deterministic noise depends on the hypothesis set, and deterministic noise decreases as the hypothesis set becomes more complex

There are two factors here: the level of deterministic noise, and the complexity of the hypothesis set used to fit the data. Both of these factors are affected by having the target fixed while expanding the hypothesis set. The two effects push overfitting in different directions.

If you want to see the impact of deterministic noise by itself, without having another factor that affects overfitting in play, you should fix the hypothesis set and increase the complexity of the target function.

rohanag 05-12-2012 02:44 AM

Re: doubt in lecture 11, deterministic noise
 
Quote:

Originally Posted by yaser (Post 2060)
If you want to see the impact of deterministic noise by itself, without having another factor that affects overfitting in play, you should fix the hypothesis set and increase the complexity of the target function.

got it, thanks Professor. So if we fix the H set and increase complexity of the target function, out overfit measure will increase.

Can we say the overfit measure, indicates noise?

yaser 05-12-2012 02:53 AM

Re: doubt in lecture 11, deterministic noise
 
Quote:

Originally Posted by rohanag (Post 2064)
got it, thanks Professor. So if we fix the H set and increase complexity of the target function, out overfit measure will increase.

Can we say the overfit measure, indicates noise?

It does. The specific way it indicates it is still a research topic.

rohanag 05-12-2012 02:56 AM

Re: doubt in lecture 11, deterministic noise
 
Quote:

Originally Posted by yaser (Post 2060)
There are two factors here: the level of deterministic noise, and the complexity of the hypothesis set used to fit the data. Both of these factors are affected by having the target fixed while expanding the hypothesis set. The two effects push overfitting in different directions.

Thanks again Professor, I have just one more question.

The two effects push overfitting in different directions, this means that if we fix a target, and expand H, the more complex H has a better ability to tackle the deterministic noise, but the complexity of the H in turn, makes for bad overfitting?

yaser 05-12-2012 03:03 AM

Re: doubt in lecture 11, deterministic noise
 
Quote:

Originally Posted by rohanag (Post 2066)
Thanks again Professor, I have just one more question.

The two effects push overfitting in different directions, this means that if we fix a target, and expand H, the more complex H has a better ability to tackle the deterministic noise, but the complexity of the H in turn, makes for bad overfitting?

The second part is accurate as you phrased it. The first part I would rephrase by saying that the more complex {\cal H} is dealing with lower deterministic noise.

mic00 05-12-2012 02:28 PM

Re: doubt in lecture 11, deterministic noise
 
Is there anything interesting to be said about the relationship between overfitting and the difference between \bar g and the truly best hypothesis in {\cal H}? I think I've seen some others say that the deterministic noise is not dependent on the size of the data set, but I am wondering whether this difference is what accounts for the overfitting caused by deterministic noise. E.g. for complex enough target function, if our sample data set is very big, and we have a simple model (high bias/deterministic noise), we are not really overfitting, as I understand it.

Edit: I've been assuming that the definitions of bias and variance take expectations over all data sets of a particular fixed size N -- I don't think this was explicitly stated, but I also don't think it makes sense otherwise. In homework #4, I computed a value for \bar g that was very far from the best possible hypothesis (lowest mean squared error), because N was so low in that case.

menon.rx 08-20-2012 09:08 PM

Re: doubt in lecture 11, deterministic noise
 
I think I am still a bit unclear about deterministic noise. Doesn't the amount of noise (deterministic or stochastic) depend on both the bias and variance of the noise? For a given N, the more complex noise will have a higher variance but lower bias. Hence doesn't the amount of noise depend upon N and the relative complexity difference between the target function and the hypothesis (the level of deterministic noise if you will)? :clueless:

Andrs 08-21-2012 01:12 AM

Re: doubt in lecture 11, deterministic noise
 
In general a more complex H implies lower "deterministic noise" but it is important to take into consideration the amount of training data that you have (N) when discussing Eout. In the example shown in lecture 11 the target function was very complex (50th order) and the training data was noiseless. We could see that a simple hypothesis (second order pol) gave a much better Eout than the more complex hypothesis (10th order polynomial). In this case there was only "deterministic noise" and the more complex Hypothesis performed much worse even if the "deterministic noise" was lower for the more complex H.


All times are GMT -7. The time now is 02:54 PM.

Powered by vBulletin® Version 3.8.3
Copyright ©2000 - 2019, Jelsoft Enterprises Ltd.
The contents of this forum are to be used ONLY by readers of the Learning From Data book by Yaser S. Abu-Mostafa, Malik Magdon-Ismail, and Hsuan-Tien Lin, and participants in the Learning From Data MOOC by Yaser S. Abu-Mostafa. No part of these contents is to be communicated or made accessible to ANY other person or entity.