LFD Book Forum (http://book.caltech.edu/bookforum/index.php)
-   Homework 7 (http://book.caltech.edu/bookforum/forumdisplay.php?f=136)
-   -   HW 7, Q5 - a bit vague formulation (http://book.caltech.edu/bookforum/showthread.php?t=1117)

 dvs79 08-23-2012 02:14 AM

HW 7, Q5 - a bit vague formulation

Quote:
 "... values ...out-of-sample classification error obtained for the model chosen in each of the above two experiments, respectively"
.

So there are 2 questions:

1. "Chosen model" means models which we chose using validation (in Q1 and Q3)? Or models, chosen on basis of Eout (Q2 and Q4)?

2. How should we evaluate Eout here, in Q5 - just take Eout from results of Q2 and Q4? Or using chosen model, train the model on the whole D (in.dta) again and evaluate Eout on out.dta?

 yaser 08-23-2012 02:53 AM

Re: HW 7, Q5 - a bit vague formulation

Quote:
 Originally Posted by dvs79 (Post 4312) 1. "Chosen model" means models which we chose using validation (in Q1 and Q3)? Or models, chosen on basis of Eout (Q2 and Q4)? 2. How should we evaluate Eout here, in Q5 - just take Eout from results of Q2 and Q4? Or using chosen model, train the model on the whole D (in.dta) again and evaluate Eout on out.dta?
1. Chosen model is the one based on validation (Q1 and Q3).

2. Yes, take results from Q2 and Q4. No restoration of the data set in this case as stated in the preamble, so train only using the training examples (which is already done).

 dvs79 08-23-2012 03:35 AM

Re: HW 7, Q5 - a bit vague formulation

Thanks!

But still didn't get the correct answer to Q5. Did anybody?

Re: HW 7, Q5 - a bit vague formulation

I haven't submitted my answers yet but I have a problem with Q5.
My out of sample errors aren't consistent with any of the answers given, considering the choice of models, if I try to round the errors to the closest.
I have reviewed my code 10 times but can't find something wrong with it...

 Thomaseid 08-26-2012 05:32 PM

Re: HW 7, Q5 - a bit vague formulation

Quote:
 Originally Posted by AdrienL (Post 4464) I haven't submitted my answers yet but I have a problem with Q5. My out of sample errors aren't consistent with any of the answers given, considering the choice of models, if I try to round the errors to the closest. I have reviewed my code 10 times but can't find something wrong with it...
I probably have the same problem with you. I am struggling between two choices. None of them gives both numbers close to my answer...

 tzs29970 08-26-2012 10:19 PM

Re: HW 7, Q5 - a bit vague formulation

Some things to check:
• Are you sure you are testing the right models? It would be easy for an off by one error when picking components of your model to mess up your results.
• Make sure you are splitting the data into training and validation sets correctly. In particular, for the second set some people have been confused over whether the training set is the first 10 points or the last 10 points.
• Are you perhaps using linear regression code you wrote for last week's problems? Did you write that code to default to doing regularization, since we were doing regularization last week? The problem this week didn't say to use regularization.

I'm pretty sure that if the code is right, there will be no doubt which answers to choose.

Re: HW 7, Q5 - a bit vague formulation

Quote:
 Originally Posted by tzs29970 (Post 4475) Some things to check: [...]Make sure you are splitting the data into training and validation sets correctly. In particular, for the second set some people have been confused over whether the training set is the first 10 points or the last 10 points. [...]
Wow, I just rechecked the homework formulation and indeed I think I messed that up. The training for the second part (Q3 and Q4) is supposed to be on the last ten points (the same ones we used to validate on the first questions), right ? Whereas the validation will be the first 25 ones, that is the ones we used in Q1 and Q2 we used to train.

If that's what I messed up, then thanks a lot :) !

(I will try again tonight and let you know if it fixes my issues.)

Re: HW 7, Q5 - a bit vague formulation

Thanks tzs, indeed you found my problem !

 Thomaseid 08-27-2012 10:02 AM

Re: HW 7, Q5 - a bit vague formulation

Quote:
 Originally Posted by tzs29970 (Post 4475) [LIST][*]Make sure you are splitting the data into training and validation sets correctly. In particular, for the second set some people have been confused over whether the training set is the first 10 points or the last 10 points.
Indeed, I used the first 10 points for training, and it should be the last 10. Thank you so much.

 chrischi 08-27-2012 12:51 PM

Re: HW 7, Q5 - a bit vague formulation

Of course I used the wrong 10 in the 2nd training, too :D.