LFD Book Forum  

Go Back   LFD Book Forum > Course Discussions > Online LFD course > Homework 7

Reply
 
Thread Tools Display Modes
  #1  
Old 08-21-2012, 07:42 AM
Andrs Andrs is offline
Member
 
Join Date: Jul 2012
Posts: 47
Default Using the whole Data lec13

In the lecture 13 (Validation),
The 10-Fold cross validation mechanism with the training data D is used to select the best "learning model" . My question is if there is any point in running the selected hypothesis (best_Hypotheses in the selected model) using the whole training data set (D) in order to get a better estimate of Eout . Or is the Ecv (cross validation Error) a good enough estimate of Eout.
Reply With Quote
  #2  
Old 08-21-2012, 12:06 PM
yaser's Avatar
yaser yaser is offline
Caltech
 
Join Date: Aug 2009
Location: Pasadena, California, USA
Posts: 1,477
Default Re: Using the whole Data lec13

Quote:
Originally Posted by Andrs View Post
In the lecture 13 (Validation),
The 10-Fold cross validation mechanism with the training data D is used to select the best "learning model" . My question is if there is any point in running the selected hypothesis (best_Hypotheses in the selected model) using the whole training data set (D) in order to get a better estimate of Eout . Or is the Ecv (cross validation Error) a good enough estimate of Eout.
It is a good idea to restore the full data set and use it for training once the model has been selected, but the problem with using the full data set for estimating E_{\rm out} for any hypothesis in this process is that part of the data set would have already been used for training to come up with this hypothesis, so that part will have a built-in bias. The cross-validation data points, although they are fewer, do not have that bias hence their estimate of E_{\rm out} is more reliable.
__________________
Where everyone thinks alike, no one thinks very much
Reply With Quote
  #3  
Old 08-21-2012, 12:25 PM
Andrs Andrs is offline
Member
 
Join Date: Jul 2012
Posts: 47
Default Re: Using the whole Data lec13

Thanks for the quick answer.
I would like to check that I really understood your recomendation: I will be consuming all my trainning-data with the cross validation procedure. Through the CV I select the model and the hypothesis (g-) with the corresponding parameters and I get Ecv that is a good estimate of Eout.
Your suggestion is that I could use this model (hypothesis set) and (re)train it on the full trainning-data in order to select a new hypothesis(g+). This new hypothesis(g+) may do better than the hypothesis (g-) but the only safer estimate for Eout is the estimate that I got thru the cross validation(Ecv). The only "problem" here is that now I do not have any data to "test" this new hypothesis (g+).
Reply With Quote
  #4  
Old 08-21-2012, 12:52 PM
yaser's Avatar
yaser yaser is offline
Caltech
 
Join Date: Aug 2009
Location: Pasadena, California, USA
Posts: 1,477
Default Re: Using the whole Data lec13

Quote:
Originally Posted by Andrs View Post
Thanks for the quick answer.
I would like to check that I really understood your recomendation: I will be consuming all my trainning-data with the cross validation procedure. Through the CV I select the model and the hypothesis (g-) with the corresponding parameters and I get Ecv that is a good estimate of Eout.
Your suggestion is that I could use this model (hypothesis set) and (re)train it on the full trainning-data in order to select a new hypothesis(g+). This new hypothesis(g+) may do better than the hypothesis (g-) but the only safer estimate for Eout is the estimate that I got thru the cross validation(Ecv). The only "problem" here is that now I do not have any data to "test" this new hypothesis (g+).
The hypothesis trained on the full data set, denoted by g which you refer to as g+, is indeed the result of this process. To estimate its E_{\rm out}, we still use the cross validation estimate for g^-, notwithstanding the fact that it is a different hypothesis (but close enough) for the reason you outline; we have no cross validation data points left to evaluate g directly.
__________________
Where everyone thinks alike, no one thinks very much
Reply With Quote
  #5  
Old 08-21-2012, 12:58 PM
rainbow rainbow is offline
Member
 
Join Date: Jul 2012
Posts: 41
Default Re: Using the whole Data lec13

I think you summarized the idea very well. I guess the idea behind CV is to estimate the E_out (by E_cv) in situations when you are short on data to start with. Then you can't afford losing data points when you reevaluate the optimal g^- on \mathcal{D} in order to get g^+.
Reply With Quote
Reply

Tags
crossvalidation error

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT -7. The time now is 12:12 AM.


Powered by vBulletin® Version 3.8.3
Copyright ©2000 - 2019, Jelsoft Enterprises Ltd.
The contents of this forum are to be used ONLY by readers of the Learning From Data book by Yaser S. Abu-Mostafa, Malik Magdon-Ismail, and Hsuan-Tien Lin, and participants in the Learning From Data MOOC by Yaser S. Abu-Mostafa. No part of these contents is to be communicated or made accessible to ANY other person or entity.