HW 7, Q5 - a bit vague formulation

[dvs79]

It reads

"... values ...out-of-sample classification error obtained for the model chosen in each of the above two experiments, respectively"

.

So there are 2 questions:

1. "Chosen model" means models which we chose using validation (in Q1 and Q3)? Or models, chosen on basis of Eout (Q2 and Q4)?

2. How should we evaluate Eout here, in Q5 - just take Eout from results of Q2 and Q4? Or using chosen model, train the model on the whole D (in.dta) again and evaluate Eout on out.dta?

[yaser]

Originally Posted by dvs79

1. "Chosen model" means models which we chose using validation (in Q1 and Q3)? Or models, chosen on basis of Eout (Q2 and Q4)?

2. How should we evaluate Eout here, in Q5 - just take Eout from results of Q2 and Q4? Or using chosen model, train the model on the whole D (in.dta) again and evaluate Eout on out.dta?

1. Chosen model is the one based on validation (Q1 and Q3).

2. Yes, take results from Q2 and Q4. No restoration of the data set in this case as stated in the preamble, so train only using the training examples (which is already done).

[dvs79]

Thanks!


But still didn't get the correct answer to Q5. Did anybody?

[AdrienL]

I haven't submitted my answers yet but I have a problem with Q5.
My out of sample errors aren't consistent with any of the answers given, considering the choice of models, if I try to round the errors to the closest.
I have reviewed my code 10 times but can't find something wrong with it...

[Thomaseid]

Originally Posted by AdrienL

I haven't submitted my answers yet but I have a problem with Q5.
My out of sample errors aren't consistent with any of the answers given, considering the choice of models, if I try to round the errors to the closest.
I have reviewed my code 10 times but can't find something wrong with it...

I probably have the same problem with you. I am struggling between two choices. None of them gives both numbers close to my answer...

[tzs29970]

Some things to check:

• Are you sure you are testing the right models? It would be easy for an off by one error when picking components of your model to mess up your results.
• Make sure you are splitting the data into training and validation sets correctly. In particular, for the second set some people have been confused over whether the training set is the first 10 points or the last 10 points.
• Are you perhaps using linear regression code you wrote for last week's problems? Did you write that code to default to doing regularization, since we were doing regularization last week? The problem this week didn't say to use regularization.

I'm pretty sure that if the code is right, there will be no doubt which answers to choose.

[AdrienL]

Originally Posted by tzs29970

Some things to check:
[...]

• Make sure you are splitting the data into training and validation sets correctly. In particular, for the second set some people have been confused over whether the training set is the first 10 points or the last 10 points.

[...]

Wow, I just rechecked the homework formulation and indeed I think I messed that up. The training for the second part (Q3 and Q4) is supposed to be on the last ten points (the same ones we used to validate on the first questions), right ? Whereas the validation will be the first 25 ones, that is the ones we used in Q1 and Q2 we used to train.

If that's what I messed up, then thanks a lot !

(I will try again tonight and let you know if it fixes my issues.)

[AdrienL]

Thanks tzs, indeed you found my problem !

[Thomaseid]

Originally Posted by tzs29970

[LIST][*]Make sure you are splitting the data into training and validation sets correctly. In particular, for the second set some people have been confused over whether the training set is the first 10 points or the last 10 points.

Indeed, I used the first 10 points for training, and it should be the last 10. Thank you so much.

[chrischi]

Of course I used the wrong 10 in the 2nd training, too .
Thx for your help