Empirical assessment of the assumptions of ComBat with diffusion tensor imaging

Michael E. Kim, Chenyu Gao, Leon Y. Cai, Qi Yang, Nancy R. Newlin, Karthik Ramadass, Angela L. Jefferson, Derek B. Archer, Niranjana Shashikumar, Kimberly R. Pechman, Katherine A. Gifford, Timothy J. Hohman, Lori L. Beason‐Held, Susan M. Resnick, Stefan Winzeck, Kurt G. Schilling, Panpan Zhang, Daniel Moyer, Bennett A. LandmanJournal of Medical Imaging 2024 · DOI

TL;DR

This paper empirically assesses the assumptions of ComBat, a popular tool for harmonizing multi-site diffusion tensor imaging (DTI) data. They found that ComBat's reliability is sensitive to sample size and age differences between sites, and that simply checking residual distributions isn't enough to guarantee stability. The study provides practical guidelines for when ComBat can be reliably used for DTI harmonization.

What Problem Does This Paper Address?

Diffusion Tensor Imaging (DTI) is a powerful technique for studying the brain's white matter. However, DTI data collected from different scanners or acquisition sites can have significant biases, making it difficult to combine data from multiple sources for larger studies. ComBat is a widely used statistical method to correct for these "site biases" and harmonize data. While ComBat is popular, its performance and the validity of its underlying assumptions, especially for DTI data, haven't been thoroughly investigated across various real-world scenarios like differing sample sizes, age distributions, and site imbalances. This paper aims to fill that gap by empirically testing ComBat's robustness under these varying conditions.

Methods

The researchers created a "silver standard" dataset by carefully matching participants from two DTI datasets (BLSA and VMAP) based on demographics. They then used a bootstrapping approach to simulate various multi-site scenarios. This involved creating numerous subsets of the silver standard cohort with different total sample sizes, varying degrees of sample size imbalance between sites, and different mean age differences. For each simulated scenario, they applied ComBat to harmonize mean fractional anisotropy (FA) and mean diffusivity (MD) values. They then evaluated ComBat's reliability by comparing its results to the "silver standard" and by assessing key metrics like the estimated effect of age on FA (BAGE), and the estimated site-shift ($\hat{\gamma}_{sf}$) and site-scale ($\hat{\delta}_{sf}$) parameters. They also checked if ComBat's underlying assumptions (like normality of residuals) were violated under different conditions.

Datasets

The study utilized two existing DTI datasets:

  • Baltimore Longitudinal Study of Aging (BLSA)
  • Vanderbilt Memory and Aging Project (VMAP)

These datasets were combined to create a "silver standard" cohort of 358 participants, matched for cognitive status, sex, and age within 4 years. This "silver standard" served as a benchmark for evaluating ComBat's performance under simulated multi-site conditions.

Key Insights and Ideas

  • Sample Size Matters (A Lot): ComBat's reliability significantly decreases with smaller sample sizes per site. The study suggests a threshold of around N > 162 for stable FA harmonization and N > 308 for MD harmonization.
  • Age Differences Can Be Tricky: Larger mean age differences between sites can also lead to instability in ComBat's harmonization, particularly for FA. A conservative threshold of a 2-year age difference is recommended for stability.
  • Residual Normality Isn't the Whole Story: While ComBat assumes normally distributed residuals, the study found that checking this assumption alone isn't sufficient to predict ComBat's stability. In some cases, residuals might appear normal even when ComBat is performing poorly.
  • Bootstrapping is Key: The authors advocate for a bootstrapping methodology, similar to their approach, to assess ComBat's reliability for a specific dataset, rather than relying solely on checking model assumptions.
  • DTI Data's Nature: DTI data, often having "heavy tails," may require larger sample sizes for ComBat to remain stable compared to other data types.

Key Results

  • ComBat harmonization of mean FA values becomes unstable at sample sizes below N < 162.
  • For mean MD, ComBat remains stable for sample sizes N > 162, indicating different DTI scalars have varying sensitivities.
  • A mean age difference of less than 4 years is generally well-behaved for BAGE estimation.
  • For site-specific bias estimates ($\hat{\gamma}_{sf}$ and $\hat{\delta}_{sf}$), stability thresholds were identified: around N > 252 for sample size, a 9:10 imbalance ratio, and a 4-year covariate shift.
  • ComBat's performance was found to be similar to other linear models (OLS and LME) when harmonizing DTI data, especially in scenarios with relatively stable conditions.

Limitations and Future Work

  • Continuous vs. Categorical Covariates: The study focused on age as a continuous covariate shift. Future work could explore the impact of categorical covariate shifts or shifts involving multiple covariates.
  • Number of Sites: The analysis primarily focused on two sites. Harmonizing data from three or more sites might reveal different error patterns.
  • B-value Correction: The study did not investigate the effect of prior b-value correction on ComBat's performance, which could be a valuable area for future research, especially given the different b-values used in the study datasets.
  • Generalizability: While the study provides guidelines, the authors caution that these may not be universally applicable and that each dataset might require careful examination.

The authors suggest that future work could also examine ComBat's reliability with the same total sample size but with more sites and smaller sample sizes per site.

Summary updated

Abstract

Prior to harmonization of DTI data with ComBat, the input cohort should be examined for size and covariate distributions of each site. Direct assessment of residual distributions is less informative on stability than bootstrap analysis. We caution use ComBat of in situations that do not conform to the above thresholds.
BibTeX
@article{kim2024empirical,
  author = {Michael E. Kim and Chenyu Gao and Leon Y. Cai and Qi Yang and Nancy R. Newlin and Karthik Ramadass and Angela L. Jefferson and Derek B. Archer and Niranjana Shashikumar and Kimberly R. Pechman and Katherine A. Gifford and Timothy J. Hohman and Lori L. Beason‐Held and Susan M. Resnick and Stefan Winzeck and Kurt G. Schilling and Panpan Zhang and Daniel Moyer and Bennett A. Landman},
  title = {{Empirical assessment of the assumptions of ComBat with diffusion tensor imaging}},
  journal = {Journal of Medical Imaging},
  year = {2024},
  doi = {10.1117/1.jmi.11.2.024011},
}
Figure