View Single Post
  #7  
Old 07-14-2014, 09:45 AM
BojanVujatovic BojanVujatovic is offline
Member
 
Join Date: Jan 2013
Posts: 13
Default Re: consistency issue on page 65

It is somewhat late but I'd like to give analytic solution for E_{out}, bias and var and the verification of values here. Firstly,

\begin{aligned}
\overline{g}(x) &= E_{\mathcal{D}}[g^{(\mathcal{D})}(x)] =  E_{x_1, x_2}[g^{(x_1, x_2)}(x)]=  E_{x_1, x_2}[a^{(x_1, x_2)}x +b^{(x_1, x_2)}] \\
&= E_{x_1, x_2}[a^{(x_1, x_2)}]x + E_{x_1, x_2}[b^{(x_1, x_2)}] = \overline{a}x + \overline{b}
\end{aligned}

a^{(x_1, x_2)} and b^{(x_1, x_2)} are parameters we get after minimising squared error function on some x_1, x_2:

\begin{aligned}
\left(a^{(x_1, x_2)}, b^{(x_1, x_2)}\right) =  \underset{a, b \in \mathbb{R}}{\text{argmin }}E_{in}(a, b)=  \underset{a, b \in \mathbb{R}}{\text{argmin }}\dfrac{1}{2} \displaystyle\sum_{i=1}^2 \left(ax_i + b - \sin \pi x_i \right)^2
\end{aligned}

and we can get them by solving the following system of equations (condition for extreme value of function):

\begin{cases}
\dfrac{\partial}{\partial a} E_{in}(a, b) = 0 \\ \\
\dfrac{\partial}{\partial b} E_{in}(a, b) = 0
\end{cases}

The solution is: \left(a^{(x_1, x_2)}, b^{(x_1, x_2)}\right) = \left(\dfrac{\sin \pi x_1 - sin \pi x_2}{x_1-x_2}, \dfrac{-x_2\sin \pi x_1 + x_1\sin \pi x_2}{x_1-x_2} \right)


Now,

\begin{aligned}
\overline{a} = E_{x_1, x_2}[a^{(x_1, x_2)}] &= \int\limits_{-1}^{-1} \int\limits_{-1}^{-1}  a^{(x_1, x_2)} P[x_1][x_2] \,\text{d}x_1 \,\text{d}x_2 \\
&= \int\limits_{-1}^{-1} \int\limits_{-1}^{-1}  \dfrac{\sin \pi x_1 - sin \pi x_2}{x_1-x_2} \cdot \dfrac{1}{2} \cdot \dfrac{1}{2} \cdot\,\text{d}x_1 \,\text{d}x_2 \approx  0.7759
\end{aligned}

\begin{aligned}
\overline{b} = E_{x_1, x_2}[b^{(x_1, x_2)}] &= \int\limits_{-1}^{-1} \int\limits_{-1}^{-1}  b^{(x_1, x_2)} P[x_1][x_2] \,\text{d}x_1 \,\text{d}x_2 \\
&=  \int\limits_{-1}^{-1} \int\limits_{-1}^{-1} \dfrac{-x_2\sin \pi x_1 + x_1\sin \pi x_2}{x_1-x_2} \cdot \dfrac{1}{2} \cdot \dfrac{1}{2} \cdot\,\text{d}x_1 \,\text{d}x_2 =  0
\end{aligned}

So, \overline{g}(x) =  \overline{a}x + \overline{b}\approx 0.7759x

Now we can calculate all the terms:

\begin{aligned}
\text{bias} &= E_x\left[\left(\overline{g}(x) - f(x) \right)^2 \right] = \int\limits_{-1}^{-1}\left(\overline{a}x + \overline{b} - \sin \pi x \right)^2 P[x] \,\text{d}x \\
&\approx \int\limits_{-1}^{-1}\left(0.7759x - \sin \pi x \right)^2 \cdot \dfrac{1}{2} \cdot \,\text{d}x \approx 0.2069
\end{aligned}


\begin{aligned}
\text{var} &= E_x\left[E_{\mathcal{D}}\left[\left(\overline{g}(x) - g^{\mathcal{D}}(x) \right)^2 \right]\right] = E_{x, x_1, x_2}\left[\left(\overline{a}x + \overline{b} -a^{(x_1, x_2)}x +b^{(x_1, x_2)} \right)^2 \right] 
\end{aligned}
\begin{aligned}
 &= \int\limits_{-1}^{-1}\int\limits_{-1}^{-1}\int\limits_{-1}^{-1}\left(\overline{a}x + \overline{b} -a^{(x_1, x_2)}x -b^{(x_1, x_2)} \right)^2 P[x]P[x_1]P[x_2] \,\text{d}x\,\text{d}x_1\,\text{d}x_2
\end{aligned}
\begin{aligned}
 &\approx \int\limits_{-1}^{-1}\int\limits_{-1}^{-1}\int\limits_{-1}^{-1}\left(0.7759x -\dfrac{\sin \pi x_1 - sin \pi x_2}{x_1-x_2}x -\dfrac{-x_2\sin \pi x_1 + x_1\sin \pi x_2}{x_1-x_2} \right)^2 \dfrac{1}{2^3}\,\text{d}x\,\text{d}x_1\,\text{d}x_2 \\
&\approx 1.676
\end{aligned}



\begin{aligned}
E_{\mathcal{D}}[E_{out}(g^{(\mathcal{D})})] &= E_x\left[E_{\mathcal{D}}\left[\left(\overline{g}(x) - f(x) \right)^2 \right]\right] = E_{x, x_1, x_2}\left[\left(\overline{a}x + \overline{b} -f(x) \right)^2 \right] 
\end{aligned}
\begin{aligned}
 &= \int\limits_{-1}^{-1}\int\limits_{-1}^{-1}\int\limits_{-1}^{-1}\left(\overline{a}x + \overline{b} -\sin \pi x \right)^2 P[x]P[x_1]P[x_2] \,\text{d}x\,\text{d}x_1\,\text{d}x_2
\end{aligned}
\begin{aligned}
 &\approx \int\limits_{-1}^{-1}\int\limits_{-1}^{-1}\int\limits_{-1}^{-1}\left(0.7759x -\sin \pi x \right)^2 \dfrac{1}{2^3}\,\text{d}x\,\text{d}x_1\,\text{d}x_2 \approx 1.883
\end{aligned}

So, we see that the following holds:

\begin{aligned}
E_{\mathcal{D}}[E_{out}(g^{(\mathcal{D})})] &= \text{bias} + \text{var}\\
1.883 &\approx 0.2067 + 1.676
\end{aligned}
Reply With Quote