Selection bias in the reported performances of AD classification pipelines

A. Mendelson, María A. Zuluaga, Marco Lorenzi, Brian F. Hutton, Sébastien OurselinNeuroImage 2016 · DOI

TL;DR

This paper investigates "selection bias" in Alzheimer's disease (AD) classification research, where promising results from many tested pipelines are more likely to be published, leading to inflated performance estimates. The authors propose a novel resampling method to quantify this bias, finding it can account for over half of the reported performance gains, especially in smaller datasets. This highlights a critical issue for clinical translation, as biased results could lead to ineffective diagnostic tools.

What Problem Does This Paper Address?

The field of neuroimaging, particularly in diagnosing Alzheimer's disease (AD), has seen a surge in the development of supervised learning pipelines. These pipelines aim to classify individuals into categories like healthy controls (HC) versus AD patients, or to predict disease progression in those with mild cognitive impairment (MCI). As researchers develop and test numerous pipelines, there's a natural tendency to publish only those that show the most impressive results. This "selection bias" can lead to an overestimation of a pipeline's true performance when applied to new, unseen data. The problem is that this inflated performance might not generalize to real-world clinical settings, potentially leading to the adoption of ineffective diagnostic tools and, in the worst case, harm to patients. The authors aim to quantify this bias and understand its contributing factors.

Methods

The core of the paper's methodology is a novel resampling-based experiment design. This design aims to simulate the process of researchers optimizing pipeline specifications and selecting the best performers. Here's how it works:

  1. Data Splitting: The available dataset is split into two disjoint subsets: a "left" subset and a "right" subset.
  2. Cross-Validation (CV) on Subsets: For each subset, a repeated K-fold cross-validation is performed on a set of candidate pipelines. This yields two independent performance estimates for each pipeline (one from the left subset, one from the right).
  3. Ranking and Bias Estimation: Pipelines are ranked based on their performance in the "left" subset (this is the "in-sample" estimate). The difference between the in-sample estimate and the corresponding performance estimate from the "right" subset (the "out-of-sample" estimate) provides an estimate of the selection bias for that particular rank.
  4. Symmetry and Averaging: This process is repeated with the roles of the left and right subsets reversed. The average of the bias estimates from both directions is taken to improve stability.
  5. Replication: This entire procedure is repeated thousands of times with different random partitions to obtain a robust estimate of the selection bias.

The authors also explore the effects of sample size, CV strategy (e.g., number of folds and repetitions), and the number of candidate pipelines considered on this bias.

Datasets

The study utilizes data from the Alzheimer's Disease Neuroimaging Initiative (ADNI) database. This is a well-established, large-scale dataset commonly used in AD research. The authors specifically used T1-weighted structural MRI images from the baseline time-point. They selected 1437 subjects for processing, ultimately yielding specific numbers for different classification tasks:

  • 372 Healthy Controls (HC)
  • 252 Alzheimer's Disease (AD) subjects
  • 230 Stable Mild Cognitive Impairment (MCI) subjects (MCIs)
  • 135 Progressive Mild Cognitive Impairment (MCIp) subjects

No new benchmark datasets were introduced; the focus was on analyzing existing data through a novel experimental design.

Key Insights and Ideas

  • Selection Bias is Pervasive: The study provides strong empirical evidence that selection bias is a significant issue in AD classification research, not just a theoretical concern.
  • Bias Inflates Performance: The bias can account for a substantial portion (often over 50%) of the apparent performance improvements reported for optimized pipelines, particularly in smaller sample sizes.
  • Small Samples Exacerbate Bias: Smaller datasets lead to higher variance in performance estimates, which in turn amplifies selection bias. This can create a paradoxical effect where smaller samples might appear to yield better in-sample performance due to greater bias.
  • MCI Prognosis is More Susceptible: The task of predicting MCI progression is shown to be more susceptible to selection bias than AD detection, likely due to its inherent difficulty.
  • CV Strategy Matters: While more repetitions in K-fold CV generally reduce bias, the trade-off between the number of folds and repetitions is crucial for computational efficiency and bias mitigation. Lower variance CV strategies (more repetitions, fewer folds) are recommended.
  • "True Progress" vs. "Apparent Progress": The paper clearly distinguishes between the genuine improvement in a pipeline's performance (out-of-sample) and the inflated performance observed due to selection (in-sample). A significant portion of "progress" reported in the literature might be illusory.

Key Results

  • Quantifiable Bias: The authors quantified selection bias across various experimental settings. For instance, in MCI prognosis with smaller sample sizes, the bias could be as high as 16% difference in in-sample performance compared to a 6% difference in out-of-sample performance.
  • Sample Size Impact: Bias was found to be approximately proportional to the inverse square root of the sample size. Larger sample sizes consistently led to lower bias and more reliable performance estimates.
  • CV Strategy Impact: Using more repetitions in K-fold CV (e.g., 8x3cv vs. 1x3cv) significantly reduced selection bias. Strategies with a higher number of repetitions and fewer folds (e.g., 2x12cv, 4x6cv) were more effective at reducing bias than those with fewer repetitions and more folds, given similar computational costs.
  • Literature Comparison: An analysis of published AD classification studies revealed a negative correlation between sample size and reported accuracy, suggesting tentative evidence of selection bias in the literature. Studies with smaller sample sizes and more complex pipelines (more adjustable parameters) were flagged as having higher risk factors for bias.

Limitations and Future Work

  • Simulation, Not Reality: The authors acknowledge that their experiment is a simulation and not a perfect replication of the entire AD classification research field. Implementing all published methods precisely is technically infeasible.
  • Simplified Methods: They used a limited set of relatively simple methods for registration, segmentation, and feature selection, which may not fully capture the complexity of all published pipelines.
  • No Measurement Uncertainty: The reported bias estimates do not come with measures of statistical uncertainty, which would require more complex generative models.
  • Future Work: The authors suggest that a "selection bias-free" validation strategy would involve treating the entire pipeline specification as a parameter to be selected using nested cross-validation. They also highlight the value of classification challenges (like CADDementia) for providing unbiased performance estimates. They emphasize the need for researchers to be aware of and control for selection bias in their own work and in the interpretation of published results. The findings are also relevant to other medical imaging applications beyond AD.

Summary updated

Abstract

The last decade has seen a great proliferation of supervised learning pipelines for individual diagnosis and prognosis in Alzheimer's disease. As more pipelines are developed and evaluated in the search for greater performance, only those results that are relatively impressive will be selected for publication. We present an empirical study to evaluate the potential for optimistic bias in classification performance results as a result of this selection. This is achieved using a novel, resampling-based experiment design that effectively simulates the optimisation of pipeline specifications by individuals or collectives of researchers using cross validation with limited data. Our findings indicate that bias can plausibly account for an appreciable fraction (often greater than half) of the apparent performance improvement associated with the pipeline optimisation, particularly in small samples. We discuss the consistency of our findings with patterns observed in the literature and consider strategies for bias reduction and mitigation.
BibTeX
@inproceedings{mendelson2016selection,
  author = {A. Mendelson and María A. Zuluaga and Marco Lorenzi and Brian F. Hutton and Sébastien Ourselin},
  title = {{Selection bias in the reported performances of AD classification pipelines}},
  booktitle = {NeuroImage},
  year = {2016},
  doi = {10.1016/j.nicl.2016.12.018},
}
Figure