Quantifying Confounding Bias in Neuroimaging Datasets with Causal Inference

Christian Wachinger, Benjamín Gutiérrez-Becker, Anna Rieckmann, Sebastian PölsterlarXiv 2019 · arXiv

Quantifying Confounding Bias in Neuroimaging Datasets with Causal Inference

Introduction

As neuroimaging moves toward "big data," researchers increasingly pool MRI scans from multiple independent studies to achieve the sample sizes required for complex machine learning models. However, these datasets are collected with varying goals, demographics, inclusion criteria, and scanner hardware. This heterogeneity introduces severe biases: selection bias (e.g., recruiting specific age groups), acquisition bias (e.g., different MRI manufacturers), and processing bias. If a machine learning model is trained on this pooled data, it risks learning these biases rather than true, generalizable biological signals. Furthermore, it is difficult to determine if a statistical correlation is a true biological cause or if it is confounded by an unmeasured third variable, such as scanner type.

Methodology

The authors address this problem through a two-phase approach: detection and quantification.

Detection: "Name That Dataset"

To detect bias, the authors trained a Random Forest classifier to predict the source dataset of a scan using only FreeSurfer-derived physical measurements (brain volume and cortical thickness). In an unbiased, harmonized environment, the classifier should perform near random chance.

Quantification: Causal Inference via MDL

To systematically study bias, the authors proposed a causal inference framework to distinguish between a causal model (biological factors $X$ directly influence brain measure $Y$) and a confounded model (latent confounders $Z$ influence both $X$ and $Y$).

Causal Directed Acyclic Graphs (DAGs) representing different relationships between biological factors (X), brain measurements (Y), and latent confounders (Z).

The framework relies on the algorithmic Markov condition, which posits that the true generative process is the one with the lowest Kolmogorov complexity. This is approximated using Minimum Description Length (MDL). The causal model is represented by linear regression, while the confounded model is represented by probabilistic Principal Component Analysis (PCA). By calculating the difference in MDL between these models, the authors can quantify the extent of confounding without identifying the specific confounder.

Datasets

The study utilized a massive pooled dataset of 12,207 MRI T1 brain scans from 15 large-scale public datasets, including ABIDE I & II, ADHD200, ADNI, HCP, and others. All scans were processed uniformly using FreeSurfer version 5.3 to extract volume and thickness metrics, with only one scan per subject retained to avoid longitudinal data leakage.

Key Results

The results highlight significant hidden biases in pooled neuroimaging data.

Classification accuracy and confusion matrix for identifying the source dataset from brain volume and thickness measurements.

Dataset Classification

The Random Forest classifier achieved 73.3% accuracy in predicting the source dataset based on volume and thickness, far exceeding the random chance level of 6.7%. Even using only age and sex yielded 42.2% accuracy, proving that while demographics play a role, the high accuracy is primarily driven by the image-derived measurements themselves. This suggests that the scanner and study design leave an indelible "fingerprint" on the data.

Causal vs. Confounded Relationships

Analysis of the relationship between demographics (age, sex) and brain volume revealed:

  • High Causality: The Human Connectome Project (HCP), which uses a single dedicated scanner and strict protocols, showed the strongest evidence of true causal relationships.
  • High Confounding: Heterogeneous datasets (e.g., COBRE, HBN, PPMI) that pool data from multiple sites or have loose inclusion criteria showed significant confounding.
  • Biological Plausibility: In the ADNI dataset, the MDL method correctly identified the expansion of lateral ventricles as highly causal—a known biological reality of aging—while flagging cerebellum cortex variations as heavily confounded.

Discussion and Limitations

While this framework successfully detects and quantifies bias, it does not yet provide a method to remove it. The results are also tied to the specific mathematical models chosen (linear regression and PCA). Furthermore, the authors fixed the dimensionality of the unobserved confounder to $k=1$, whereas real-world confounding is likely multidimensional. Future work may extend this framework to estimate the causal effects of technical parameters, such as the impact of magnetic field strength on signal-to-noise ratios.

Summary updated

Abstract

Neuroimaging datasets keep growing in size to address increasingly complex medical questions. However, even the largest datasets today alone are too small for training complex machine learning models. A potential solution is to increase sample size by pooling scans from several datasets. In this work, we combine 12,207 MRI scans from 15 studies and show that simple pooling is often ill-advised due to introducing various types of biases in the training data. First, we systematically define these biases. Second, we detect bias by experimentally showing that scans can be correctly assigned to their respective dataset with 73.3% accuracy. Finally, we propose to tell causal from confounding factors by quantifying the extent of confounding and causality in a single dataset using causal inference. We achieve this by finding the simplest graphical model in terms of Kolmogorov complexity. As Kolmogorov complexity is not directly computable, we employ the minimum description length to approximate it. We empirically show that our approach is able to estimate plausible causal relationships from real neuroimaging data.
BibTeX
@misc{wachinger2019quantifying,
  author = {Christian Wachinger and Benjamín Gutiérrez-Becker and Anna Rieckmann and Sebastian Pölsterl},
  title = {{Quantifying Confounding Bias in Neuroimaging Datasets with Causal Inference}},
  year = {2019},
  eprint = {1907.04102},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1907.04102},
}
Figure