View Single Post
  #3  
Old 05-20-2013, 08:34 AM
Elroch Elroch is offline
Invited Guest
 
Join Date: Mar 2013
Posts: 143
Default Re: Criss-cross validation

Thanks, Prof. Lin, that is indeed relevant. Am I right that everywhere later in the paper that 10-fold cross-validation is referred to, it means 10 x (10-fold cross-validation) ?

The instability of LOO cross-validation is interesting. I see two reasons for this instability. The first is that the individual runs only differ on two data points, so the hypotheses generated may be highly correlated. (This is likely to be mitigated by higher correlation with the hypothesis generated when finally using all the data). In addition, there is no scope for error reduction with LOO by repeating the cross validation and averaging errors. So, in your study, LOO errors were only based on 90 OOS observations, compared with 10 x 90 in the 10 x 10-fold cross-validation. What is your experience on the nature of the instability?

I observe that in a circumstance where the second reason is the important one, "Leave-two-out" cross-validation provides the potential to improve this enormously, with up to 90*89 OOS observations. The question is how much damage is done as a result of the correlation between hypotheses generated with data that differs on only 2 points (as well as those differing on 4 points).

Another observation is that it is perfectly reasonable to combine cross-validations with different fractions of data, again in order to reduce noise in the cross-validation procedure.
Reply With Quote