Exercise 3.4

xuewei4d · #1 06-14-2013, 10:08 AM

I didn't get correct answer to Exercise 3.4 (c).

Exercise 3.4(b), I think the answer would be $E_{\text{in}}(\mathbf{w}_{\text{in}})=\frac{1}{N}\epsilon^TH\epsilon$

Exercise 3.4(c), by independence between different $\epsilon_i$ , I have $\mathbb{E}_{\mathcal{D}}[E_{\text{in}}(\mathbf{w}_{\text{in}})] = \frac{1}{N} \sum_i H_{ii} \mathbb{E}(\epsilon^2) = \frac{\sigma^2}{N}(d+1)$ .

Where am I wrong?

htlin · #2 06-17-2013, 08:27 PM

You can consider double-checking your answer of 3.4(b). Hope this helps.

i_need_some_help · #3 10-06-2013, 02:16 PM

I am not sure how to approach part (a). Are we supposed to explain why that in-sample estimate intuitively makes sense, or (algebraically) manipulate expressions given earlier into it?

magdon · #4 10-06-2013, 09:18 PM

Algebraically manipulate earlier expressions and you should get 3.4(a). It is essentially a restatement of $\hat{\mathbf{y}}=X{\mathbf{w}}_{lin}$ .

Sweater Monkey · #5 10-07-2013, 12:00 AM

I'm not sure if I'm going about part (e) correctly.

I'm under the impression that $E_{\text{test}}(\mathbf{w}_{\text{lin}})=\frac{1}{N}||X{\mathbf{w}}_{lin}-\mathbf{y'}||^2$

where $\hat{\mathbf{y}}=X{\mathbf{w}}_{lin}=X\mathbf{w}^*+H\mathbf{\epsilon}$ as derived earlier
and $\mathbf{y'}=\mathbf{w}^{*T}\mathbf{x}+\mathbf{\epsilon'}=X\mathbf{w}^*+\mathbf{\epsilon'}$

This lead me to $\frac{1}{N}||H\mathbf{\epsilon}-\mathbf{\epsilon'}||^2$

I carried out the expansion of this expression and then simplified into the relevant terms but my final answer is $\sigma^2(1+(d+1))$ because the N term cancels out.

Am I starting out correctly up until this expansion or is my thought process off from the start? And if I am heading in the right direction is there any obvious reason that I may be expanding the expression incorrectly? Any help would be greatly appreciated.

ddas2 · #6 10-07-2013, 01:46 AM

1. I got $y^{\prime}=y-\epsilon+\epsilon^{\prime}$.
and $\hat{y}-y^{\prime}=H\epsilon +\epsilon^{\prime}$.

magdon · #7 10-07-2013, 05:53 AM

You got it mostly right. Your error is assuming both term, the H term and the one without the H give an N to cancel the N in the denominator. One term gives an N and the other gives a (d+1).

Quote:

Originally Posted by Sweater Monkey

I'm not sure if I'm going about part (e) correctly.

I'm under the impression that $E_{\text{test}}(\mathbf{w}_{\text{lin}})=\frac{1}{N}||X{\mathbf{w}}_{lin}-\mathbf{y'}||^2$

where $\hat{\mathbf{y}}=X{\mathbf{w}}_{lin}=X\mathbf{w}^*+H\mathbf{\epsilon}$ as derived earlier
and $\mathbf{y'}=\mathbf{w}^{*T}\mathbf{x}+\mathbf{\epsilon'}=X\mathbf{w}^*+\mathbf{\epsilon'}$

This lead me to $\frac{1}{N}||H\mathbf{\epsilon}-\mathbf{\epsilon'}||^2$

I carried out the expansion of this expression and then simplified into the relevant terms but my final answer is $\sigma^2(1+(d+1))$ because the N term cancels out.

Am I starting out correctly up until this expansion or is my thought process off from the start? And if I am heading in the right direction is there any obvious reason that I may be expanding the expression incorrectly? Any help would be greatly appreciated.

Sweater Monkey · #8 10-07-2013, 09:09 AM

Quote:

Originally Posted by magdon

You got it mostly right. Your error is assuming both term, the H term and the one without the H give an N to cancel the N in the denominator. One term gives an N and the other gives a (d+1).

Yes I realize that only one term should have the N so the issue must be in how I'm expanding the expression.

I think my problem is how I'm looking at the trace of the $\mathbf{\epsilon}^T\mathbf{\epsilon}$ matrix.

I'm under the impression that $\mathbf{\epsilon}^T\mathbf{\epsilon}$ produces an NxN matrix with a diagonal of all $\sigma^2$ values and 0 elsewhere. I come to this conclusion because the $\epsilon$ are all independent so when multiplied together the covariance of any two should be zero while the covariance of any $\epsilon_i\epsilon_i$ should be the variance of $\sigma^2$ . So then the trace of this matrix should have a sum along the diagonal of $N\sigma^2$ , shouldn't it?

aaoam · #9 10-07-2013, 09:18 AM

I'm having a bit of difficulty with 3.4b. I take \hat(y) - y and multiply by (XX^T)^{-1}XX^T, which ends up reducing the expression to just H\epsilon. However, then I can't use 3.3c in simplifying 3.3c, which makes me think I did something wrong. Can somebody give me a pointer?

Also, it'd be great if there was instructions somewhere about how to post in math mode. Perhaps I just missed them?

magdon · #10 10-07-2013, 09:19 AM

Yes, that is right. You have to be more careful but use similar reasoning with

$\epsilon^TH\epsilon$

Quote:

Originally Posted by Sweater Monkey

Yes I realize that only one term should have the N so the issue must be in how I'm expanding the expression.

I think my problem is how I'm looking at the trace of the $\mathbf{\epsilon}^T\mathbf{\epsilon}$ matrix.

I'm under the impression that $\mathbf{\epsilon}^T\mathbf{\epsilon}$ produces an NxN matrix with a diagonal of all $\sigma^2$ values and 0 elsewhere. I come to this conclusion because the $\epsilon$ are all independent so when multiplied together the covariance of any two should be zero while the covariance of any $\epsilon_i\epsilon_i$ should be the variance of $\sigma^2$ . So then the trace of this matrix should have a sum along the diagonal of $N\sigma^2$ , shouldn't it?

Thread Tools
Show Printable Version Email this Page
Display Modes
Linear Mode Switch to Hybrid Mode Switch to Threaded Mode