doubt in lecture 11, deterministic noise

[rohanag]

In the lecture 11, it is mentioned that deterministic noise depends on the hypothesis set, and deterministic noise decreases as the hypothesis set becomes more complex, because the set can tackle some of the deterministic noise.
But the experiment performed on slide 7 of lecture 11 suggests that the more complex model fails badly when compared to the less complex model.

The above statements seem to contradict in my mind, I cannot put them together. Can anyone please clear my doubt. Thank you.

[AqibEjaz]

This is indeed confusing and after spending some time thinking about this point I think I have finally understood it (I hope). Deterministic noise is nothing but the bias in the modelling hypothesis. So the more complex model will indeed have less deterministic noise (small bias). But this does not imply that this model will also have smaller Eout. Because Eout also depends on the variance of the hypothesis and since the variance of the more complex model will be large for small N, this means that Eout will be large for more complex model. But if we have sufficiently large sample size (i.e. large N) then both the variance and the bias (i.e. deterministic noise) will be small for complex model. Hence in this case the more complex model will outperform the simpler model. So the lesson learnt is: Complex model is better than simple model provided we have sufficient data. For small data sets, complex models overfit and it is better to choose simple models.

[yaser]

Originally Posted by rohanag

In the lecture 11, it is mentioned that deterministic noise depends on the hypothesis set, and deterministic noise decreases as the hypothesis set becomes more complex

There are two factors here: the level of deterministic noise, and the complexity of the hypothesis set used to fit the data. Both of these factors are affected by having the target fixed while expanding the hypothesis set. The two effects push overfitting in different directions.

If you want to see the impact of deterministic noise by itself, without having another factor that affects overfitting in play, you should fix the hypothesis set and increase the complexity of the target function.

[rohanag]

Originally Posted by yaser

If you want to see the impact of deterministic noise by itself, without having another factor that affects overfitting in play, you should fix the hypothesis set and increase the complexity of the target function.

got it, thanks Professor. So if we fix the H set and increase complexity of the target function, out overfit measure will increase.

Can we say the overfit measure, indicates noise?

[yaser]

Originally Posted by rohanag

got it, thanks Professor. So if we fix the H set and increase complexity of the target function, out overfit measure will increase.

Can we say the overfit measure, indicates noise?

It does. The specific way it indicates it is still a research topic.

[rohanag]

Originally Posted by yaser

There are two factors here: the level of deterministic noise, and the complexity of the hypothesis set used to fit the data. Both of these factors are affected by having the target fixed while expanding the hypothesis set. The two effects push overfitting in different directions.

Thanks again Professor, I have just one more question.

The two effects push overfitting in different directions, this means that if we fix a target, and expand H, the more complex H has a better ability to tackle the deterministic noise, but the complexity of the H in turn, makes for bad overfitting?

[yaser]

Originally Posted by rohanag

Thanks again Professor, I have just one more question.

The two effects push overfitting in different directions, this means that if we fix a target, and expand H, the more complex H has a better ability to tackle the deterministic noise, but the complexity of the H in turn, makes for bad overfitting?

The second part is accurate as you phrased it. The first part I would rephrase by saying that the more complex is dealing with lower deterministic noise.

[mic00]

Is there anything interesting to be said about the relationship between overfitting and the difference between and the truly best hypothesis in ? I think I've seen some others say that the deterministic noise is not dependent on the size of the data set, but I am wondering whether this difference is what accounts for the overfitting caused by deterministic noise. E.g. for complex enough target function, if our sample data set is very big, and we have a simple model (high bias/deterministic noise), we are not really overfitting, as I understand it.

Edit: I've been assuming that the definitions of bias and variance take expectations over all data sets of a particular fixed size -- I don't think this was explicitly stated, but I also don't think it makes sense otherwise. In homework #4, I computed a value for that was very far from the best possible hypothesis (lowest mean squared error), because was so low in that case.

[menon.rx]

I think I am still a bit unclear about deterministic noise. Doesn't the amount of noise (deterministic or stochastic) depend on both the bias and variance of the noise? For a given N, the more complex noise will have a higher variance but lower bias. Hence doesn't the amount of noise depend upon N and the relative complexity difference between the target function and the hypothesis (the level of deterministic noise if you will)?

[Andrs]

In general a more complex H implies lower "deterministic noise" but it is important to take into consideration the amount of training data that you have (N) when discussing Eout. In the example shown in lecture 11 the target function was very complex (50th order) and the training data was noiseless. We could see that a simple hypothesis (second order pol) gave a much better Eout than the more complex hypothesis (10th order polynomial). In this case there was only "deterministic noise" and the more complex Hypothesis performed much worse even if the "deterministic noise" was lower for the more complex H.