REQUEST: Q7 How-To

galo · #1 01-27-2016, 11:50 PM

Can anybody share the rundown to get the answer? I can't grasp a way to do it. Plus it gets frustrating because I feel like it's more of an algebra problem rather than a Machine Learning one.

tddevlin · #2 08-14-2016, 03:56 PM

**Spoiler Alert: This post contains the full solution**

First let's make sure we have the right picture.

So

and

are sitting on the

-axis, while

is somewhere to the right of the

-axis at height 1. For this dataset, leave-one-out validation entails fitting our model to two of the points, then testing the fit on the third. Let's start with the constant model,

. When we fit this model on two data points,

will simply be the average of the

-coordinates of the two points.

Leaving out, we find . The error is .
Leaving out, we also find . Again, .
Finally, leaving out, . The error is .

The overall cross-validation error is the average of the three individual errors, $E_{\text{val}}(h_0) = \frac{e_1 + e_2 + e_3}{3} = 1/2$ , as you can verify. Looking ahead, we would like to find the value of $\rho$ that makes $E_{\text{val}}(h_1) = 1/2$ .

Let's turn to the linear model,

. The easy case is when

is left out. The resulting fitted line is simply

and the error is

.

Things get more complicated when

is left out. We need to find the equation of the line through

and

. Using slope-intercept form and rearranging, you can check that the fitted line has slope equal to its intercept, $a = b = \frac{1}{1 + \rho}$ . The error on

is $e_2 = (h_1(x_2) - y_2)^2 = (ax_2 + b - 0)^2 = \left( \frac{2}{1+\rho} \right)^2$ .

A similar derivation yields $e_3 = \left( \frac{-2}{\rho - 1} \right)^2$ .

Putting it all together gives us $E_{\text{val}}(h_1) = \frac{1}{3} \left[ 1 + \left( \frac{2}{1+\rho} \right)^2+ \left( \frac{-2}{\rho - 1} \right)^2 \right]$ . If we set this equal to 1/2 (the error from the constant model), we have a quadratic equation in one unknown, which we can solve using the quadratic formula (alternatively, dumping the whole equation into WolframAlpha gives you the roots directly).

Hope that helped!

Thread Tools
Show Printable Version Email this Page
Display Modes
Linear Mode Switch to Hybrid Mode Switch to Threaded Mode