Detect and Correct Bias in Multi-Site Neuroimaging Datasets

Christian Wachinger, Anna Rieckmann, Sebastian Pölsterl

TL;DR

Pooling neuroimaging data from multiple studies is essential for training powerful machine learning models, but it introduces severe "dataset bias" from differing scanner hardware and protocols. This paper demonstrates that a classifier can guess which dataset a brain scan came from with an alarming 71.5% accuracy, and introduces ComBat++, a novel harmonization method inspired by genetics that removes these hidden site effects while preserving crucial biological signals.

What Problem Does This Paper Address?

To train complex deep learning models or achieve high statistical power, neuroimaging researchers increasingly rely on "big data" by pooling MRI scans from multiple independent studies. However, simply throwing all these scans into a single bucket is highly problematic.

Different studies use different scanner manufacturers, magnetic field strengths, imaging protocols, and patient recruitment criteria. This introduces selection and measurement biases—collectively known as dataset bias. Instead of learning true neuroanatomical patterns associated with a disease, machine learning models often inadvertently learn to identify the specific site or study where the scan was acquired. Identifying and correcting these latent confounders without erasing the actual biological variation (like age-related brain changes) remains a massive hurdle in multi-site neuroimaging.

Methods

The authors tackle dataset bias through a three-pronged approach:

  1. Name That Dataset: To prove bias exists, they train a Random Forest classifier to predict the source dataset of a scan based solely on image-derived features (such as regional brain volumes and cortical thickness).
  2. Causal Inference with Latent Confounders: To determine if a statistical relationship is truly causal ($X \rightarrow Y$) or merely confounded by an unobserved variable ($X \leftarrow Z \rightarrow Y$), they use an algorithmic framework based on Kolmogorov complexity. By approximating complexity via the Minimum Description Length (MDL) principle, they can identify the simplest, most plausible graphical model factorization without needing to explicitly measure the hidden confounder $Z$.
  3. ComBat++ (Extended Harmonization): Standard data harmonization techniques like ComBat adjust for known site-specific additive and multiplicative effects. The authors propose an extension—ComBat++—that accounts for unknown global variations. Borrowing a concept from genome-wide association studies (GWAS) called population stratification, they compute Principal Components (PCs) across all image features to capture and remove generic, non-biological scanner variation.

Datasets

The study represents one of the largest mega-analyses to date, aggregating 35,320 T1-weighted structural MRI brain scans across 17 large-scale public datasets, encompassing 210 unique imaging sites.

The datasets include:

  • ABCD (Adolescent Brain Cognitive Development)
  • ABIDE I & II (Autism Brain Imaging Data Exchange)
  • ADHD200
  • ADNI (Alzheimer's Disease Neuroimaging Initiative)
  • AIBL (Australian Imaging Biomarkers and Lifestyle flagship study of ageing)
  • COBRE, CORR, GSP, HBN, HCP, IXI, MCIC, NKI, OASIS, PPMI, and UKB (UK Biobank).

All scans were uniformly processed using FreeSurfer (v5.3) to extract 55 volume measures and 70 cortical thickness measures.

Key Insights and Ideas

  • Brain Scans Secretly Encode the Scanner: Image-derived metrics like cortical thickness are supposed to reflect pure anatomy, but they implicitly carry strong signatures of the imaging site's hardware and settings.
  • The Harmonization Paradox: Harmonization is a delicate balancing act. Naive methods like Z-score normalization successfully wipe out dataset-specific signatures, but they also accidentally erase vital biological signals, such as the natural correlation between brain atrophy and aging.
  • Cross-Pollination from Genetics: Treating unknown scanner variations similarly to "population stratification" in genetics allows researchers to use global principal components as proxy variables to filter out unobserved environmental noise.

Key Results

1. Proving Dataset Bias

In the "Name That Dataset" experiment, using a combination of brain volume and thickness measures, the classifier achieved 71.5% accuracy in identifying the source dataset (where random chance given the 17 datasets would be roughly 7.8%). This confirms that site-specific signatures are heavily embedded in the data.

Figure 2: Left: Dataset classification accuracy for age and sex, volume, thickness, and their combination. The percentage of the data used for training is shown in log-scale. Lines show the average score over 50 repetitions, error bars show the standard deviation. Right: Confusion matrix for volume and thickness with 70% training data.

2. Mitigating Bias with ComBat++

When applying ComBat++ (incorporating principal components to remove latent variation), the dataset classification accuracy dropped significantly to 45%. Crucially, unlike standard normalization, ComBat++ actually strengthened the expected biological correlations with age—preserving the fact that lateral ventricles expand and the hippocampus shrinks as people get older.

3. Improving Downstream Tasks

The authors evaluated brain age prediction using a leave-one-dataset-out cross-validation scheme. Harmonizing the data with ComBat++ significantly reduced the Mean Absolute Error (MAE) of the age predictions compared to using raw, unharmonized measures.

Limitations and Future Work

While ComBat++ effectively mitigates dataset bias, the authors note that aggressive harmonization still carries the risk of removing relevant subject-specific information if the target variables are not carefully protected during the regression step.

Additionally, the causal inference framework deployed in this study assumes a linear relationship between continuous variables. The authors suggest that future work could incorporate more advanced, non-linear causal models—such as Gaussian Process regression or non-linear PCA—to better capture the brain's complex, non-linear structural trajectories.

Summary updated

Abstract

The desire to train complex machine learning algorithms and to increase the statistical power in association studies drives neuroimaging research to use ever-larger datasets. The most obvious way to increase sample size is by pooling scans from independent studies. However, simple pooling is often ill-advised as selection, measurement, and confounding biases may creep in and yield spurious correlations. In this work, we combine 35,320 magnetic resonance images of the brain from 17 studies to examine bias in neuroimaging. In the first experiment, Name That Dataset, we provide empirical evidence for the presence of bias by showing that scans can be correctly assigned to their respective dataset with 71.5% accuracy. Given such evidence, we take a closer look at confounding bias, which is often viewed as the main shortcoming in observational studies. In practice, we neither know all potential confounders nor do we have data on them. Hence, we model confounders as unknown, latent variables. Kolmogorov complexity is then used to decide whether the confounded or the causal model provides the simplest factorization of the graphical model. Finally, we present methods for dataset harmonization and study their ability to remove bias in imaging features. In particular, we propose an extension of the recently introduced ComBat algorithm to control for global variation across image features, inspired by adjusting for population stratification in genetics. Our results demonstrate that harmonization can reduce dataset-specific information in image features. Further, confounding bias can be reduced and even turned into a causal relationship. However, harmonziation also requires caution as it can easily remove relevant subject-specific information. Code is available at https://github.com/ai-med/Dataset-Bias.
BibTeX
@article{wachinger2021detect,
  author = {Christian Wachinger and Anna Rieckmann and Sebastian Pölsterl},
  title = {{Detect and Correct Bias in Multi-Site Neuroimaging Datasets}},
  journal = {Medical Image Analysis},
  year = {2021},
  doi = {10.1016/j.media.2020.101879},
}
Figure