LFD Book Forum - View Single Post

BojanVujatovic · #7 07-14-2014, 08:45 AM

It is somewhat late but I'd like to give analytic solution for $E_{out}$ , bias and var and the verification of values here. Firstly,

$\begin{aligned} \overline{g}(x) &= E_{\mathcal{D}}[g^{(\mathcal{D})}(x)] = E_{x_1, x_2}[g^{(x_1, x_2)}(x)]= E_{x_1, x_2}[a^{(x_1, x_2)}x +b^{(x_1, x_2)}] \\ &= E_{x_1, x_2}[a^{(x_1, x_2)}]x + E_{x_1, x_2}[b^{(x_1, x_2)}] = \overline{a}x + \overline{b} \end{aligned}$

$a^{(x_1, x_2)}$ and $b^{(x_1, x_2)}$ are parameters we get after minimising squared error function on some

:

$\begin{aligned} \left(a^{(x_1, x_2)}, b^{(x_1, x_2)}\right) = \underset{a, b \in \mathbb{R}}{\text{argmin }}E_{in}(a, b)= \underset{a, b \in \mathbb{R}}{\text{argmin }}\dfrac{1}{2} \displaystyle\sum_{i=1}^2 \left(ax_i + b - \sin \pi x_i \right)^2 \end{aligned}$

and we can get them by solving the following system of equations (condition for extreme value of function):

$\begin{cases} \dfrac{\partial}{\partial a} E_{in}(a, b) = 0 \\ \\ \dfrac{\partial}{\partial b} E_{in}(a, b) = 0 \end{cases}$

The solution is: $\left(a^{(x_1, x_2)}, b^{(x_1, x_2)}\right) = \left(\dfrac{\sin \pi x_1 - sin \pi x_2}{x_1-x_2}, \dfrac{-x_2\sin \pi x_1 + x_1\sin \pi x_2}{x_1-x_2} \right)$

Now,

$\begin{aligned} \overline{a} = E_{x_1, x_2}[a^{(x_1, x_2)}] &= \int\limits_{-1}^{-1} \int\limits_{-1}^{-1} a^{(x_1, x_2)} P[x_1][x_2] \,\text{d}x_1 \,\text{d}x_2 \\ &= \int\limits_{-1}^{-1} \int\limits_{-1}^{-1} \dfrac{\sin \pi x_1 - sin \pi x_2}{x_1-x_2} \cdot \dfrac{1}{2} \cdot \dfrac{1}{2} \cdot\,\text{d}x_1 \,\text{d}x_2 \approx 0.7759 \end{aligned}$

$\begin{aligned} \overline{b} = E_{x_1, x_2}[b^{(x_1, x_2)}] &= \int\limits_{-1}^{-1} \int\limits_{-1}^{-1} b^{(x_1, x_2)} P[x_1][x_2] \,\text{d}x_1 \,\text{d}x_2 \\ &= \int\limits_{-1}^{-1} \int\limits_{-1}^{-1} \dfrac{-x_2\sin \pi x_1 + x_1\sin \pi x_2}{x_1-x_2} \cdot \dfrac{1}{2} \cdot \dfrac{1}{2} \cdot\,\text{d}x_1 \,\text{d}x_2 = 0 \end{aligned}$

So, $\overline{g}(x) = \overline{a}x + \overline{b}\approx 0.7759x$

Now we can calculate all the terms:

$\begin{aligned} \text{bias} &= E_x\left[\left(\overline{g}(x) - f(x) \right)^2 \right] = \int\limits_{-1}^{-1}\left(\overline{a}x + \overline{b} - \sin \pi x \right)^2 P[x] \,\text{d}x \\ &\approx \int\limits_{-1}^{-1}\left(0.7759x - \sin \pi x \right)^2 \cdot \dfrac{1}{2} \cdot \,\text{d}x \approx 0.2069 \end{aligned}$

$\begin{aligned} \text{var} &= E_x\left[E_{\mathcal{D}}\left[\left(\overline{g}(x) - g^{\mathcal{D}}(x) \right)^2 \right]\right] = E_{x, x_1, x_2}\left[\left(\overline{a}x + \overline{b} -a^{(x_1, x_2)}x +b^{(x_1, x_2)} \right)^2 \right] \end{aligned}$
$\begin{aligned} &= \int\limits_{-1}^{-1}\int\limits_{-1}^{-1}\int\limits_{-1}^{-1}\left(\overline{a}x + \overline{b} -a^{(x_1, x_2)}x -b^{(x_1, x_2)} \right)^2 P[x]P[x_1]P[x_2] \,\text{d}x\,\text{d}x_1\,\text{d}x_2 \end{aligned}$
$\begin{aligned} &\approx \int\limits_{-1}^{-1}\int\limits_{-1}^{-1}\int\limits_{-1}^{-1}\left(0.7759x -\dfrac{\sin \pi x_1 - sin \pi x_2}{x_1-x_2}x -\dfrac{-x_2\sin \pi x_1 + x_1\sin \pi x_2}{x_1-x_2} \right)^2 \dfrac{1}{2^3}\,\text{d}x\,\text{d}x_1\,\text{d}x_2 \\ &\approx 1.676 \end{aligned}$

$\begin{aligned} E_{\mathcal{D}}[E_{out}(g^{(\mathcal{D})})] &= E_x\left[E_{\mathcal{D}}\left[\left(\overline{g}(x) - f(x) \right)^2 \right]\right] = E_{x, x_1, x_2}\left[\left(\overline{a}x + \overline{b} -f(x) \right)^2 \right] \end{aligned}$
$\begin{aligned} &= \int\limits_{-1}^{-1}\int\limits_{-1}^{-1}\int\limits_{-1}^{-1}\left(\overline{a}x + \overline{b} -\sin \pi x \right)^2 P[x]P[x_1]P[x_2] \,\text{d}x\,\text{d}x_1\,\text{d}x_2 \end{aligned}$
$\begin{aligned} &\approx \int\limits_{-1}^{-1}\int\limits_{-1}^{-1}\int\limits_{-1}^{-1}\left(0.7759x -\sin \pi x \right)^2 \dfrac{1}{2^3}\,\text{d}x\,\text{d}x_1\,\text{d}x_2 \approx 1.883 \end{aligned}$

So, we see that the following holds:

$\begin{aligned} E_{\mathcal{D}}[E_{out}(g^{(\mathcal{D})})] &= \text{bias} + \text{var}\\ 1.883 &\approx 0.2067 + 1.676 \end{aligned}$