LFD Book Forum  

Go Back   LFD Book Forum > Course Discussions > Online LFD course > Homework 7

Reply
 
Thread Tools Display Modes
  #1  
Old 08-23-2012, 03:14 AM
dvs79 dvs79 is offline
Member
 
Join Date: Jul 2012
Location: Moscow, Russia
Posts: 24
Default HW 7, Q5 - a bit vague formulation

It reads
Quote:
"... values ...out-of-sample classification error obtained for the model chosen in each of the above two experiments, respectively"
.

So there are 2 questions:

1. "Chosen model" means models which we chose using validation (in Q1 and Q3)? Or models, chosen on basis of Eout (Q2 and Q4)?

2. How should we evaluate Eout here, in Q5 - just take Eout from results of Q2 and Q4? Or using chosen model, train the model on the whole D (in.dta) again and evaluate Eout on out.dta?
Reply With Quote
  #2  
Old 08-23-2012, 03:53 AM
yaser's Avatar
yaser yaser is offline
Caltech
 
Join Date: Aug 2009
Location: Pasadena, California, USA
Posts: 1,477
Default Re: HW 7, Q5 - a bit vague formulation

Quote:
Originally Posted by dvs79 View Post
1. "Chosen model" means models which we chose using validation (in Q1 and Q3)? Or models, chosen on basis of Eout (Q2 and Q4)?

2. How should we evaluate Eout here, in Q5 - just take Eout from results of Q2 and Q4? Or using chosen model, train the model on the whole D (in.dta) again and evaluate Eout on out.dta?
1. Chosen model is the one based on validation (Q1 and Q3).

2. Yes, take results from Q2 and Q4. No restoration of the data set in this case as stated in the preamble, so train only using the training examples (which is already done).
__________________
Where everyone thinks alike, no one thinks very much
Reply With Quote
  #3  
Old 08-23-2012, 04:35 AM
dvs79 dvs79 is offline
Member
 
Join Date: Jul 2012
Location: Moscow, Russia
Posts: 24
Default Re: HW 7, Q5 - a bit vague formulation

Thanks!


But still didn't get the correct answer to Q5. Did anybody?
Reply With Quote
  #4  
Old 08-26-2012, 01:15 PM
AdrienL AdrienL is offline
Member
 
Join Date: Jul 2012
Posts: 13
Default Re: HW 7, Q5 - a bit vague formulation

I haven't submitted my answers yet but I have a problem with Q5.
My out of sample errors aren't consistent with any of the answers given, considering the choice of models, if I try to round the errors to the closest.
I have reviewed my code 10 times but can't find something wrong with it...
Reply With Quote
  #5  
Old 08-26-2012, 06:32 PM
Thomaseid Thomaseid is offline
Junior Member
 
Join Date: Jul 2012
Posts: 6
Default Re: HW 7, Q5 - a bit vague formulation

Quote:
Originally Posted by AdrienL View Post
I haven't submitted my answers yet but I have a problem with Q5.
My out of sample errors aren't consistent with any of the answers given, considering the choice of models, if I try to round the errors to the closest.
I have reviewed my code 10 times but can't find something wrong with it...
I probably have the same problem with you. I am struggling between two choices. None of them gives both numbers close to my answer...
Reply With Quote
  #6  
Old 08-26-2012, 11:19 PM
tzs29970 tzs29970 is offline
Invited Guest
 
Join Date: Apr 2012
Posts: 52
Default Re: HW 7, Q5 - a bit vague formulation

Some things to check:
  • Are you sure you are testing the right models? It would be easy for an off by one error when picking components of your model to mess up your results.
  • Make sure you are splitting the data into training and validation sets correctly. In particular, for the second set some people have been confused over whether the training set is the first 10 points or the last 10 points.
  • Are you perhaps using linear regression code you wrote for last week's problems? Did you write that code to default to doing regularization, since we were doing regularization last week? The problem this week didn't say to use regularization.

I'm pretty sure that if the code is right, there will be no doubt which answers to choose.
Reply With Quote
  #7  
Old 08-27-2012, 02:46 AM
AdrienL AdrienL is offline
Member
 
Join Date: Jul 2012
Posts: 13
Default Re: HW 7, Q5 - a bit vague formulation

Quote:
Originally Posted by tzs29970 View Post
Some things to check:
[...]
  • Make sure you are splitting the data into training and validation sets correctly. In particular, for the second set some people have been confused over whether the training set is the first 10 points or the last 10 points.
[...]
Wow, I just rechecked the homework formulation and indeed I think I messed that up. The training for the second part (Q3 and Q4) is supposed to be on the last ten points (the same ones we used to validate on the first questions), right ? Whereas the validation will be the first 25 ones, that is the ones we used in Q1 and Q2 we used to train.

If that's what I messed up, then thanks a lot !

(I will try again tonight and let you know if it fixes my issues.)
Reply With Quote
  #8  
Old 08-27-2012, 10:27 AM
AdrienL AdrienL is offline
Member
 
Join Date: Jul 2012
Posts: 13
Default Re: HW 7, Q5 - a bit vague formulation

Thanks tzs, indeed you found my problem !
Reply With Quote
  #9  
Old 08-27-2012, 11:02 AM
Thomaseid Thomaseid is offline
Junior Member
 
Join Date: Jul 2012
Posts: 6
Default Re: HW 7, Q5 - a bit vague formulation

Quote:
Originally Posted by tzs29970 View Post
[LIST][*]Make sure you are splitting the data into training and validation sets correctly. In particular, for the second set some people have been confused over whether the training set is the first 10 points or the last 10 points.
Indeed, I used the first 10 points for training, and it should be the last 10. Thank you so much.
Reply With Quote
  #10  
Old 08-27-2012, 01:51 PM
chrischi chrischi is offline
Junior Member
 
Join Date: Jul 2012
Posts: 1
Default Re: HW 7, Q5 - a bit vague formulation

Of course I used the wrong 10 in the 2nd training, too .
Thx for your help
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT -7. The time now is 02:26 PM.


Powered by vBulletin® Version 3.8.3
Copyright ©2000 - 2019, Jelsoft Enterprises Ltd.
The contents of this forum are to be used ONLY by readers of the Learning From Data book by Yaser S. Abu-Mostafa, Malik Magdon-Ismail, and Hsuan-Tien Lin, and participants in the Learning From Data MOOC by Yaser S. Abu-Mostafa. No part of these contents is to be communicated or made accessible to ANY other person or entity.