Re: Noise
At a high level, noise is the descrapancy between the best you could do within (not the learned hypothesis) and . You cannot model stochastic noise. In a similar way, cannot model the deterministic noise.
At a high level, to a first approximation, the deterministic noise 'level' is quantified by the bias term. However effect of the noise does not end there. When there is noise, it is also harder to find the best fit. This shows up as the indirect impact of the noise, which is in the var term. Yes, the bias is determined by the mean hypothesis. For most standard models, this is close to the best fit, but not necessarily so as you point out. With respect to thinking about deterministic noise, it is better to think about the actual best fit, and the part of that is 'orthogonal' to this best fit acts like noise and cannot be modeled. Yes you are correct. The fact that the data set is finite and random is not related to the stochastic noise. It is not the randomness of the data set per se that is bad, but the finiteness of it. So you are right, it may be a good idea to emphasize that the randomness in the data set is not related to the stochastic noise. In fact we had at some point toyed with introducing the term 'finite sample noise' to highlight this point, but decided against it. However, this randomness of the finite data set is very crucial because that is actually what leads to the var term. If the data set were large, tending to infinity, then the var term would tend to zero (typically at a rate of 1/N). So what is actually going on is as follows. There is stochastic noise and deterministic noise. These have direct impact on the error through the and bias terms. Now there is also the var term, which is the indirect impact of the stochastic and deterministic noise. This indirect impact is due to the 'not being able to find the best fit' from a finite data set. It is hard to find the best fit from the finite data set, and even harder when there is noise (stochastic or deterministic). Quote:

All times are GMT 7. The time now is 09:45 PM. 
Powered by vBulletin® Version 3.8.3
Copyright ©2000  2020, Jelsoft Enterprises Ltd.
The contents of this forum are to be used ONLY by readers of the Learning From Data book by Yaser S. AbuMostafa, Malik MagdonIsmail, and HsuanTien Lin, and participants in the Learning From Data MOOC by Yaser S. AbuMostafa. No part of these contents is to be communicated or made accessible to ANY other person or entity.