Lec11: Overfitting in terms of (bias, var, stochasticnoise)
We have:
totalnoise =
var (overfittingnoise ? )
+ bias (deterministicnoise)
+ stochasticnoise
Qs:
1. Is overfittingnoise the var part alone? From Prof’s lecture, I tend to conclude that it is var caused because of attempt to fit stochasticnoise i.e. overfittingnoise really is an interplay of (stochasticnoise > variance). Need help in interpreting it.
2. When we try to arrest the overfitting, using brakes(regularization) and/or validation, are we really working with overfitting alone ?
In case of validation, we will have a measure of totalerror : Is it that the relativity of totalerrors across choice of modelcomplexity(e.g. H2 Vs H10), is giving us an estimate of relative measure of overfitting across choices of hypothesiscomplexity?
In case of brakes(regularization) : will the brake really be applied on overfitting alone, and not other parts of totalerror, esp bias part ?
3. Consider a case in which targetcomplexity is 2nd order polynomial and we chose a 2nd order(H2) and a 10th order polynomial(H10) to fit it. How will the overfit and bias vary for the two hypothesis (as N grows on the xaxis)?
Specifically, will the H10 have overfitting (with or without stochastic noise)? Also, H10 should have higher bias compared to H2 ?
4. Is there a notion of underfitting wrt TargetFunction ? When we try to fit a 10th order polynomial targetfunction, with a 2nd order polynomial hypothesis, are we not underfitting ? If so, can we associate underfitting to bias then ? If not, what else ?
Thanks
