Cross-validation failure: Small sample sizes lead to large error bars

Gaël Varoquaux

Introduction

In the field of neuroimaging, researchers have increasingly moved away from traditional statistical testing toward predictive modeling—using machine learning to decode mental states from fMRI or predict clinical outcomes. To validate these models, cross-validation (CV) has become the gold standard, where data is split into training and testing sets to assess generalization. However, this approach faces a significant "power failure" at the sample sizes typical in brain imaging (N < 100). While researchers often assume a high cross-validated accuracy score indicates a robust effect, this paper demonstrates that such results are frequently plagued by massive, systematically underestimated error bars.

Methodology and Datasets

The study employs a three-pronged approach to expose these failure modes:

  1. Empirical Analysis: Re-evaluating existing neuroimaging datasets (fMRI, MEG, and Voxel-Based Morphometry) to measure discrepancies between CV-estimated accuracy and true generalization on large independent sets.
  2. Simulations: Using synthetic Gaussian data to show that even under ideal conditions, small samples inherently produce high variance in CV results.
  3. Meta-Analysis: Reviewing hundreds of published studies to correlate reported accuracy with sample size, looking for evidence of publication bias.

Data sources include classic fMRI datasets like Haxby et al. (2001), the Kaggle MLSP 2014 Schizophrenia challenge, and aggregated data from major review papers covering Autism, Depression, and Alzheimer’s.

The Uncertainty of Small Samples

A central finding is the "±10% Rule of Thumb": for a typical study with 100 samples, the error bar on prediction accuracy is roughly ±10%. This means a reported 75% accuracy could easily reflect a true accuracy anywhere between 65% and 85%.

Estimation errors in empirical neuroimaging data (a) and the Kaggle competition (d) reveal wide distributions of error, particularly in small-sample regimes.

Furthermore, the common practice of using the Standard Error of the Mean (SEM) across CV folds significantly underestimates true uncertainty—often by a factor of 2 to 4—because training sets in different folds overlap and are not independent. This high variance leads to "vibration effects," where minor choices in the analytic pipeline (e.g., smoothing levels) can swing accuracy by 10-20%, allowing for unintentional "p-hacking."

Simulations (b) illustrate how the estimation error on prediction accuracy decreases as the number of available samples increases, highlighting the instability of results in low-N studies.

Key Results and Publication Bias

The paper provides sobering confidence bounds for binary classification:

  • N = 30: Confidence bounds are ±15%.
  • N = 100: Confidence bounds are ±10%.
  • N = 1000: Confidence bounds shrink to ±3%.

Meta-analysis reveals a clear trend: studies with smaller samples report much higher accuracies. This suggests a "filter effect" in publishing, where low-N studies are only successful in reaching publication when they happen to hit a "lucky" high-accuracy noise spike. In one simulation with meaningless labels, varying the decoder and feature selection alone could "achieve" up to 71% accuracy on a dataset where the true chance level was 50%.

Implications for the Field

While acquiring large neuroimaging datasets is expensive, the author argues that the solution lies in changing scientific practices rather than ignoring the noise. Key recommendations include:

  • Data Sharing and Pooling: Collaborations like the UK Biobank are essential to reach the N > 1000 threshold needed for reliable results.
  • Preregistration: Deciding on analysis pipelines before data inspection to combat vibration effects.
  • Independent Validation: Shifting focus from single CV scores to testing models across multiple independent datasets.

Ultimately, until sample sizes increase significantly, small-scale brain decoding results should be treated with substantial skepticism.

Summary updated

Abstract

Predictive models ground many state-of-the-art developments in statistical brain image analysis: decoding, MVPA, searchlight, or extraction of biomarkers. The principled approach to establish their validity and usefulness is cross-validation, testing prediction on unseen data. Here, I would like to raise awareness on error bars of cross-validation, which are often underestimated. Simple experiments show that sample sizes of many neuroimaging studies inherently lead to large error bars, eg ± 10 % for 100 samples. The standard error across folds strongly underestimates them. These large error bars compromise the reliability of conclusions drawn with predictive models, such as biomarkers or methods developments where, unlike with cognitive neuroimaging MVPA approaches, more samples cannot be acquired by repeating the experiment across many subjects. Solutions to increase sample size must be investigated, tackling possible increases in heterogeneity of the data.
BibTeX
@article{varoquaux2018cross,
  author = {Gaël Varoquaux},
  title = {{Cross-validation failure: Small sample sizes lead to large error bars}},
  journal = {NeuroImage},
  year = {2018},
  doi = {10.1016/j.neuroimage.2017.06.061},
}
Figure