consistency issue on page 65

dudefromdayton · #1 04-28-2012, 07:33 PM

Example 2.8, the target function is sin(pi*x). But both target function graph labels in the second figure show sin(x) instead. Someone in the know should see that the graphed function, its label, and the target function coincide.

yaser · #2 04-28-2012, 07:40 PM

Quote:

Originally Posted by dudefromdayton

Example 2.8, the target function is sin(pi*x). But both target function graph labels in the second figure show sin(x) instead. Someone in the know should see that the graphed function, its label, and the target function coincide.

Thank you for noticing it. Indeed, it should be $\sin(\pi x)$ .

dudefromdayton · #3 04-28-2012, 08:32 PM

On the same page, I've been able to confirm the biases stated for H0 and H1, as well as the variance for H0. But for the variance of 1.69 for H1, I am obtaining 2.44 instead.

I have this problem whether I calculate the variance directly, or I calculate the out-of-sample error and subtract the bias.

It would be reassuring if I could show that my 2.44 figure is wrong, but as yet I have not succeeded.

yaser · #4 04-28-2012, 10:36 PM

Quote:

Originally Posted by dudefromdayton

On the same page, I've been able to confirm the biases stated for H0 and H1, as well as the variance for H0. But for the variance of 1.69 for H1, I am obtaining 2.44 instead.

I have this problem whether I calculate the variance directly, or I calculate the out-of-sample error and subtract the bias.

It would be reassuring if I could show that my 2.44 figure is wrong, but as yet I have not succeeded.

This is curious. To calculate $E_{\rm out}$ , you generate the two points

and

uniformly and independently, then connect them with a line. Since $y_1=\sin(\pi x_1)$ and $y_2=\sin(\pi x_2)$ , you get the equation for the line (tangent to the sine if the two points coincide). You then integrate the difference squared between the line and the sine with respect to

(times the probability density which is ${1\over 2}$ ) to get $E_{\rm out}$ for this data set, and double-integrate that with respect to

and

(times the probability density which is ${1\over 2}$ for each variable) to get the expected $E_{\rm out}$ .

Is this what you have done to come up with a numerical answer equal to $E_{\rm out}=2.44+0.21$ ?

Hint: It is much easier to Monte-Carlo.

dudefromdayton · #5 04-29-2012, 05:54 AM

My ETA having a closer look at this is 1-2 days. Schedule. I'm definitely interested in resolving this matter, and I'm happy to participate.

Update: wrapping up some computations; I'm testing an idea as to what might have happened. But the test is not quick.

dudefromdayton · #6 04-29-2012, 03:50 PM

Alas, I got bit by that old dog, numerical instability. When I run 1,000 hypotheses, I get the much higher figure I reported. 10,000 brought me a lot closer, and 100,000 has me at 1.70 variance (and presumably moving in the direction of 1.69).

These tests take a lot longer than they might, because I solve the line equation with w-a-a-y too many CPU cycles. The upside is that the same code works with several other hypothesis sets to meet a short-term need.

I feel in general like I've faced a couple of numerical stability challenges in very recent time, and it's beneficial when students have to deal with it. I imagine there might be more ahead.

BojanVujatovic · #7 07-14-2014, 08:45 AM

It is somewhat late but I'd like to give analytic solution for $E_{out}$ , bias and var and the verification of values here. Firstly,

$\begin{aligned} \overline{g}(x) &= E_{\mathcal{D}}[g^{(\mathcal{D})}(x)] = E_{x_1, x_2}[g^{(x_1, x_2)}(x)]= E_{x_1, x_2}[a^{(x_1, x_2)}x +b^{(x_1, x_2)}] \\ &= E_{x_1, x_2}[a^{(x_1, x_2)}]x + E_{x_1, x_2}[b^{(x_1, x_2)}] = \overline{a}x + \overline{b} \end{aligned}$

$a^{(x_1, x_2)}$ and $b^{(x_1, x_2)}$ are parameters we get after minimising squared error function on some

:

$\begin{aligned} \left(a^{(x_1, x_2)}, b^{(x_1, x_2)}\right) = \underset{a, b \in \mathbb{R}}{\text{argmin }}E_{in}(a, b)= \underset{a, b \in \mathbb{R}}{\text{argmin }}\dfrac{1}{2} \displaystyle\sum_{i=1}^2 \left(ax_i + b - \sin \pi x_i \right)^2 \end{aligned}$

and we can get them by solving the following system of equations (condition for extreme value of function):

$\begin{cases} \dfrac{\partial}{\partial a} E_{in}(a, b) = 0 \\ \\ \dfrac{\partial}{\partial b} E_{in}(a, b) = 0 \end{cases}$

The solution is: $\left(a^{(x_1, x_2)}, b^{(x_1, x_2)}\right) = \left(\dfrac{\sin \pi x_1 - sin \pi x_2}{x_1-x_2}, \dfrac{-x_2\sin \pi x_1 + x_1\sin \pi x_2}{x_1-x_2} \right)$

Now,

$\begin{aligned} \overline{a} = E_{x_1, x_2}[a^{(x_1, x_2)}] &= \int\limits_{-1}^{-1} \int\limits_{-1}^{-1} a^{(x_1, x_2)} P[x_1][x_2] \,\text{d}x_1 \,\text{d}x_2 \\ &= \int\limits_{-1}^{-1} \int\limits_{-1}^{-1} \dfrac{\sin \pi x_1 - sin \pi x_2}{x_1-x_2} \cdot \dfrac{1}{2} \cdot \dfrac{1}{2} \cdot\,\text{d}x_1 \,\text{d}x_2 \approx 0.7759 \end{aligned}$

$\begin{aligned} \overline{b} = E_{x_1, x_2}[b^{(x_1, x_2)}] &= \int\limits_{-1}^{-1} \int\limits_{-1}^{-1} b^{(x_1, x_2)} P[x_1][x_2] \,\text{d}x_1 \,\text{d}x_2 \\ &= \int\limits_{-1}^{-1} \int\limits_{-1}^{-1} \dfrac{-x_2\sin \pi x_1 + x_1\sin \pi x_2}{x_1-x_2} \cdot \dfrac{1}{2} \cdot \dfrac{1}{2} \cdot\,\text{d}x_1 \,\text{d}x_2 = 0 \end{aligned}$

So, $\overline{g}(x) = \overline{a}x + \overline{b}\approx 0.7759x$

Now we can calculate all the terms:

$\begin{aligned} \text{bias} &= E_x\left[\left(\overline{g}(x) - f(x) \right)^2 \right] = \int\limits_{-1}^{-1}\left(\overline{a}x + \overline{b} - \sin \pi x \right)^2 P[x] \,\text{d}x \\ &\approx \int\limits_{-1}^{-1}\left(0.7759x - \sin \pi x \right)^2 \cdot \dfrac{1}{2} \cdot \,\text{d}x \approx 0.2069 \end{aligned}$

$\begin{aligned} \text{var} &= E_x\left[E_{\mathcal{D}}\left[\left(\overline{g}(x) - g^{\mathcal{D}}(x) \right)^2 \right]\right] = E_{x, x_1, x_2}\left[\left(\overline{a}x + \overline{b} -a^{(x_1, x_2)}x +b^{(x_1, x_2)} \right)^2 \right] \end{aligned}$
$\begin{aligned} &= \int\limits_{-1}^{-1}\int\limits_{-1}^{-1}\int\limits_{-1}^{-1}\left(\overline{a}x + \overline{b} -a^{(x_1, x_2)}x -b^{(x_1, x_2)} \right)^2 P[x]P[x_1]P[x_2] \,\text{d}x\,\text{d}x_1\,\text{d}x_2 \end{aligned}$
$\begin{aligned} &\approx \int\limits_{-1}^{-1}\int\limits_{-1}^{-1}\int\limits_{-1}^{-1}\left(0.7759x -\dfrac{\sin \pi x_1 - sin \pi x_2}{x_1-x_2}x -\dfrac{-x_2\sin \pi x_1 + x_1\sin \pi x_2}{x_1-x_2} \right)^2 \dfrac{1}{2^3}\,\text{d}x\,\text{d}x_1\,\text{d}x_2 \\ &\approx 1.676 \end{aligned}$

$\begin{aligned} E_{\mathcal{D}}[E_{out}(g^{(\mathcal{D})})] &= E_x\left[E_{\mathcal{D}}\left[\left(\overline{g}(x) - f(x) \right)^2 \right]\right] = E_{x, x_1, x_2}\left[\left(\overline{a}x + \overline{b} -f(x) \right)^2 \right] \end{aligned}$
$\begin{aligned} &= \int\limits_{-1}^{-1}\int\limits_{-1}^{-1}\int\limits_{-1}^{-1}\left(\overline{a}x + \overline{b} -\sin \pi x \right)^2 P[x]P[x_1]P[x_2] \,\text{d}x\,\text{d}x_1\,\text{d}x_2 \end{aligned}$
$\begin{aligned} &\approx \int\limits_{-1}^{-1}\int\limits_{-1}^{-1}\int\limits_{-1}^{-1}\left(0.7759x -\sin \pi x \right)^2 \dfrac{1}{2^3}\,\text{d}x\,\text{d}x_1\,\text{d}x_2 \approx 1.883 \end{aligned}$

So, we see that the following holds:

$\begin{aligned} E_{\mathcal{D}}[E_{out}(g^{(\mathcal{D})})] &= \text{bias} + \text{var}\\ 1.883 &\approx 0.2067 + 1.676 \end{aligned}$

magdon · #8 07-15-2014, 07:42 AM

Thanks for the detailed analysis

Thread Tools
Show Printable Version Email this Page
Display Modes
Linear Mode Switch to Hybrid Mode Switch to Threaded Mode