Detecting and mitigating doppelgänger bias in microbiome data: impacts on machine learning and disease classification

R. Zhou, Siu Kin Ng, Joseph J.�y. Sung, Sunny H. Wong, Wilson Wen Bin GohGut 2025 · DOI

TL;DR

This paper introduces a method to identify and mitigate "doppelgänger pairs" – highly similar microbiome samples that can artificially inflate machine learning performance and bias statistical analyses. By removing these redundant samples, the authors demonstrate more reliable and biologically meaningful insights into the microbiome's role in diseases like IBD and CRC.

What Problem Does This Paper Address?

The human microbiome is a complex ecosystem crucial for health, and its dysbiosis is linked to various diseases. Analyzing microbiome data, especially for machine learning and disease classification, is challenging due to high dimensionality and sparsity. A critical, yet often overlooked, problem is the presence of "doppelgänger pairs" – samples that are highly similar to each other within the same class (e.g., two control samples that are almost identical). These pairs can act as a form of data leakage, causing machine learning models to "memorize" specific samples rather than learn generalizable patterns. This leads to overestimation of model performance, biased statistical tests, and distorted biological interpretations. This paper aims to address this by providing a standardized method to identify and remove these doppelgänger pairs, thereby improving the integrity and reliability of microbiome data analysis.

Methods

The core of the paper's methodology revolves around identifying doppelgänger pairs. This is achieved by:

  • Calculating Pairwise Pearson's Correlation Coefficients (PPCC): This quantifies the similarity between all pairs of samples.
  • Establishing a Correlation Cutoff: A conservative cutoff is determined by the maximum correlation observed between samples from different classes (e.g., a case and a control sample). Any sample pair within the same class that exceeds this cutoff is flagged as a doppelgänger pair. This strategy ensures that only biologically implausible levels of similarity within a class are identified.
  • Validation with Spearman's and Kendall's Correlations: To ensure robustness, other correlation measures were also computed.
  • Assessing Impact: The authors then evaluate the impact of these doppelgänger pairs on:
  • Machine Learning Performance: Using KNN, SVM, and Random Forest classifiers, they measure how accuracy, F1 scores, and AUC are affected by the presence and proportion of doppelgänger pairs in training and validation sets.
  • Statistical Association Tests: They use the Wilcoxon rank-sum test to assess OTU-disease associations and evaluate changes in p-values and effect size stability (using bootstrapping for log-fold change variance).
  • Network Analysis: They analyze the structural properties (average path length and clustering coefficient) of microbial co-occurrence networks before and after doppelgänger removal, using network perturbation simulations.

Datasets

The study utilized a variety of datasets to ensure generalizability:

  • MicrobiomeHD database: This comprehensive database was the primary source for various disease cohorts.
  • Specific Disease Cohorts:
    • Colorectal Cancer (CRC)
    • Inflammatory Bowel Diseases (IBD)
    • Clostridioides difficile infection (CDI)
    • Obesity
  • Inflammatory Bowel Disease Multi-omics Database (IBDMDB, also known as HMP2): This provided shotgun metagenomic data for IBD patients and controls.
  • Simulated Datasets: Synthetic datasets were generated to specifically test the doppelgänger identification method and its compatibility with different data types.

Key Insights and Ideas

  • Doppelgängers are More Prevalent in Inflammatory/Treatment-Driven Diseases: The study found a higher prevalence of doppelgänger pairs in IBD and CDI datasets compared to CRC and obesity. This suggests that conditions involving inflammation or treatment might drive microbiome convergence among patients, leading to more similar sample profiles.
  • Data Leakage is a Major Concern: The presence of doppelgänger pairs, especially when split between training and validation sets, leads to significant "data leakage." This artificially inflates model performance metrics, making models appear much better than they actually are at generalizing to new, unseen data.
  • Bias Extends Beyond ML Accuracy: Doppelgänger pairs don't just inflate ML accuracy; they also bias statistical tests by increasing false positives and decreasing effect size stability. This means that associations identified might be spurious, and the magnitude of real effects could be misrepresented.
  • Network Topology is Distorted: Microbial co-occurrence networks built from data with doppelgängers exhibit altered structural properties, potentially leading to misinterpretations of community structure and resilience.
  • Removal Uncovers Hidden Signals: Crucially, removing doppelgänger pairs not only reduces noise and false positives but can also reveal biologically meaningful signals that were previously masked by redundant data.

Key Results

  • Inflated Accuracy: Machine learning model accuracies were artificially boosted by 15-30% points across KNN, SVM, and Random Forest classifiers due to doppelgänger pairs. For example, accuracy in the CRC dataset inflated from 30-50%, CDI from 40-70%, and IBD from 85-95%.
  • Reduced Bootstrap Variance: Removing doppelgänger pairs reduced bootstrap variance in log-fold change estimates by up to 28.3% in the IBD metagenomic dataset, indicating improved stability of association results.
  • Increased Significance Estimates: In association testing, doppelgängers led to a higher number of statistically significant results. After removal, p-values for most OTUs increased, indicating a shift towards more conservative and reliable significance estimates.
  • Network Robustness: Networks constructed from doppelgänger-removed data showed improved topological stability, maintaining a longer average path length and a more gradual decline in clustering coefficient during node removal simulations.
  • Prevalence: Up to 2-10% of samples in IBD and CDI datasets were identified as doppelgänger pairs, while CRC and obesity datasets had a lower prevalence (around 1-2%).

Limitations and Future Work

  • Potential Loss of True Biological Signals: The authors acknowledge that their conservative approach of removing doppelgänger pairs could potentially lead to the loss of true biological signals.
  • Future Directions:
    • Weighting Similar Pairs: Future work could explore strategies for weighting highly similar pairs or incorporating them into model regularization rather than outright removal.
  • Multi-omics Integration: Applying the doppelgänger identification method to multi-omics datasets could provide deeper insights into the functional consequences of sample redundancy.
  • Nuanced Strategies: Developing more nuanced strategies for handling doppelgänger pairs beyond simple removal is suggested.

In essence, this study provides a crucial framework for improving the quality and reliability of microbiome data analysis, particularly for machine learning applications, by addressing the often-overlooked issue of sample redundancy.

Summary updated

Abstract

Highly similar microbiome samples - so-called "doppelgänger pairs" - can distort analysis outcomes, yet are rarely addressed in microbiome studies. Here, we demonstrate that even a small proportion of such pairs (1-10% of samples) can substantially inflate machine learning performance across diverse disease cohorts including colorectal cancer (CRC), inflammatory bowel diseases (IBD), <i>Clostridioides difficile</i> infection (CDI), and obesity. Doppelgänger pairs also bias statistical tests and distort microbial network topology. In predictive models, classification accuracy was artificially boosted by 15-30% points across KNN, SVM, and Random Forest classifiers. In association testing, doppelgängers increased false-positive rates and decreased effect size stability; their removal reduced bootstrap variance by up to 28.3%. Moreover, the removal of doppelgängers yielded more stable networks. These effects were consistently observed across 16S, shotgun metagenomic, and simulated datasets. By accounting for highly similar samples, we reduce analytical noise and false discoveries, ultimately enabling more accurate and biologically meaningful microbiome insights.
BibTeX
@inproceedings{zhou2025detecting,
  author = {R. Zhou and Siu Kin Ng and Joseph J.�Y. Sung and Sunny H. Wong and Wilson Wen Bin Goh},
  title = {{Detecting and mitigating doppelgänger bias in microbiome data: impacts on machine learning and disease classification}},
  booktitle = {Gut},
  year = {2025},
  doi = {10.1080/19490976.2025.2554196},
}
Figure