Quote:
Originally Posted by Andrew87
Hello,
I'm getting confused about . Why is it the best approximation of the target function we could obtain in the unreal case of infinite training sets ?
Thank you in advance,
Andrea

It is not necessarily the best approximation of the target function, but it is often close. If we have one, infinitesize training set, and we have infinite computational power that goes with it, we can arrive at the best approximation. In the biasvariance analysis, we are given an infinite number of
finite training sets, and we are restricted to using one of these finite training sets at a time, then averaging the resulting hypotheses. This restriction can take us away from the absolute optimal, but usually not by much.