LFD Book Forum - View Single Post

vikasatkin · #2 02-13-2013, 09:00 PM

Question: (Slide 11/23) How did you generate the polynomials? How did you choose the coefficients?

Answer: Here is the technical description of the process of generating the target function and the dataset (which may be useful, if you want to reproduce the pictures from the slide 13/23). It was actually described in the "Learning From Data" book on p.123 (section 4.1.2 "Catalysts for Overfitting").

The process of generating the target function depends on two parameters:

(degree of the generated polynomials) and $\sigma^2$ (noise level). Of course, you also need

--- amount of points in the dataset.

1. Take Legendre polynomials $P_0,\dots,P_{Q_f}$ . Note, that they are normalized according to their value at

(i.e.

), not their average square.
2. Choose coefficients $a_0,\dots,a_{Q_f}$ independently according to the standard normal distribution.
3. Generate

points (pick them randomly from

, independently from each other).
4. For every point

generate the noise $\epsilon(x_i)$ .

The target is given by $y=c\sum_{q=0}^{Q_f} a_q P_q(x) + \epsilon(x)$ . Here

is a normalization constant, which depends only on

. It is chosen in such a way, that the mean square value of $f(x)=c\sum_{q=0}^{Q_f} a_q P_q(x)$ is equal to 1 (mean with respect to both

and choices we made during this process: $\mathbb{E}_{a,x} ((f(x))^2) = 1$ ). On can compute, that
$c = \left( \sum_{q=0}^{Q_f} \frac1{2q+1} \right)^{-1/2}.$