Fig. 3

The impact of improper resampling shown on an RNASeq dataset [13, 14]. Samples are classified into Group 1 (CEU, n = 69 samples) versus group 2 (YRI, n = 60 samples) using the lasso logistic regression classifier as implemented in the glmnet package [36]. The “No CV” case did not use cross-validation to pick a value for the tuning parameter, instead using a fixed value 4e-9. The “naïve CV” method used naïve, non-nested cross-validation to pick the tuning parameter. The “nested CV” method used nested cross-validation to pick the tuning parameter, so that there was never any overlap between the data used to develop the predictor and the data used to estimate and evaluate the prediction scores. The accuracy estimated from the correct nested CV method is 95%, and from each of the other methods is 100%, the difference representing bias due to erroneous resampling