LFD Book Forum

LFD Book Forum (http://book.caltech.edu/bookforum/index.php)
-   Homework 7 (http://book.caltech.edu/bookforum/forumdisplay.php?f=136)
-   -   REQUEST: Q7 How-To (http://book.caltech.edu/bookforum/showthread.php?t=4654)

galo 01-27-2016 11:50 PM

Can anybody share the rundown to get the answer? I can't grasp a way to do it. Plus it gets frustrating because I feel like it's more of an algebra problem rather than a Machine Learning one.

tddevlin 08-14-2016 03:56 PM

Re: REQUEST: Q7 How-To
**Spoiler Alert: This post contains the full solution**

First let's make sure we have the right picture.


So P_1 and P_2 are sitting on the x-axis, while P_3 is somewhere to the right of the y-axis at height 1. For this dataset, leave-one-out validation entails fitting our model to two of the points, then testing the fit on the third. Let's start with the constant model, h_0(x) = b. When we fit this model on two data points, b will simply be the average of the y-coordinates of the two points.
  • Leaving P_1 out, we find b = 1/2. The error is e_1 = (h_0(x_1) - y_1)^2 = (b - 0)^2 = (1/2)^2.
  • Leaving P_2 out, we also find b = 1/2. Again, e_2 = (h_0(x_2) - y_2)^2 = (b - 0)^2 = (1/2)^2.
  • Finally, leaving P_3 out, b = 0. The error is e_3 = (h_0(x_3) - y_3)^2 = (0 - 1)^2 = 1.

The overall cross-validation error is the average of the three individual errors, E_{\text{val}}(h_0) = \frac{e_1 + e_2 + e_3}{3} = 1/2, as you can verify. Looking ahead, we would like to find the value of \rho that makes E_{\text{val}}(h_1) = 1/2.

Let's turn to the linear model, h_1(x) = ax + b. The easy case is when P_3 is left out. The resulting fitted line is simply y = 0 and the error is e_3 = 1.

Things get more complicated when P_2 is left out. We need to find the equation of the line through P_1 and P_3. Using slope-intercept form and rearranging, you can check that the fitted line has slope equal to its intercept, a = b = \frac{1}{1 + \rho}. The error on P_2 is e_2 = (h_1(x_2) - y_2)^2 = (ax_2 + b - 0)^2 = \left( \frac{2}{1+\rho} \right)^2.

A similar derivation yields e_3 =  \left( \frac{-2}{\rho - 1} \right)^2.

Putting it all together gives us E_{\text{val}}(h_1) = \frac{1}{3} \left[ 1 + \left( \frac{2}{1+\rho} \right)^2+ \left( \frac{-2}{\rho - 1} \right)^2 \right]. If we set this equal to 1/2 (the error from the constant model), we have a quadratic equation in one unknown, which we can solve using the quadratic formula (alternatively, dumping the whole equation into WolframAlpha gives you the roots directly).

Hope that helped!

RJT_12 12-08-2021 04:01 AM

Re: REQUEST: Q7 How-To

If we assume constant model and leave out point whit rho included i.e. (rho, 1), the mid value appears to be y axis. (mid point between (-1,0) and (1,0)).

The difference between y axis and the point (rho, 1) is rho and error term is rho^2. However, using this error for constant model does not lead to any of the answers.

However if assume error for leave out point (rho, 1) be 1, than I arrive at correct solution.

But I think rho^2 error is correct. Please help me to find where I make mistake,

All times are GMT -7. The time now is 03:57 PM.

Powered by vBulletin® Version 3.8.3
Copyright ©2000 - 2022, Jelsoft Enterprises Ltd.
The contents of this forum are to be used ONLY by readers of the Learning From Data book by Yaser S. Abu-Mostafa, Malik Magdon-Ismail, and Hsuan-Tien Lin, and participants in the Learning From Data MOOC by Yaser S. Abu-Mostafa. No part of these contents is to be communicated or made accessible to ANY other person or entity.