Assessing putative bias in prediction of anti-microbial resistance from real-world genotyping data under explicit causal assumptions

Mattia Prosperi, Christina Boucher, Jiang Bian, Simone Marini

TL;DR

This paper explores how to use causal inference techniques to fix sampling biases in machine learning models that predict antimicrobial resistance (AMR). By accounting for confounders like bacterial species, geographic location, and collection year, the authors demonstrate that "causally-informed" models are more robust and perform better when faced with real-world data shifts.

What Problem Does This Paper Address?

Predicting whether a bacteria is resistant to antibiotics (AMR) using Whole Genome Sequencing (WGS) is becoming a standard practice. Researchers typically train models on genotype-phenotype pairs from large public databases. However, these databases are not random samples; they are heavily influenced by which labs are sequencing which outbreaks in which countries.

This creates a sampling distribution shift. A model might learn that a specific genetic signature predicts resistance simply because that signature is common in a specific species that happened to be oversampled in a specific year. When you try to use that model on new data from a different country or a later time period, its accuracy often plummets. The authors aim to bridge the gap between "black-box" prediction and causal understanding to make these models more reliable.

Methods

The authors treat the presence of a genetic trait (encoded as $k$-mers) as an "exposure" and the AMR phenotype as the "outcome." They use several causal inference strategies to handle bias:

  • Causal Modeling: They developed a Partially Directed Acyclic Graph (PDAG) to map the relationships between species, country, year, genetic signatures, and AMR.
  • Propensity Score Matching (PSM): They used PSM to rebalance the dataset, effectively creating a "synthetic" randomized trial where the probability of having a genetic trait is balanced across different species and locations.
  • Covariate Adjustment: They tested models that explicitly included species, country, and year as features alongside genetic data.
  • ATE-based Feature Selection: They ranked genetic signatures ($k$-mers) based on their Average Treatment Effect (ATE) adjusted for confounders, rather than just simple correlation.

They compared two main machine learning architectures: Boosted Logistic Regression (BLR) and Random Forests (RF), evaluating them with and without these bias-handling strategies.

Datasets

The study utilized data from the Pathosystems Resource Integration Center (PATRIC), focusing specifically on resistance to the tetracycline drug class.

  • Training Set: 9,851 bacterial instances.
  • External Test Set: 1,085 instances, specifically selected from a single country (USA) and the most recent time quartile to simulate a real-world distribution shift.
  • Features: Bacterial genomes were broken down into $k$-mers (DNA fragments of length 17). Out of over 40 million unique $k$-mers, they selected the top 20,000 for analysis.

Key Insights and Ideas

  • Genetic Signatures are Proxy Confounders: $k$-mers often inherently contain information about the bacterial species. While this helps prediction, it doesn't account for how AMR prevalence changes independently by geography or time.
  • Direction vs. Magnitude: Interestingly, adjusting for bias only changed the direction of the effect for about 8.1% of genetic signatures. However, it significantly cleaned up the "noise," with 20.2% of $k$-mers losing statistical significance once confounders were accounted for.
  • Robustness to Shift: The primary benefit of causal adjustment wasn't just a higher score on a standard test set, but maintaining that performance when the model was moved to an "external" dataset with different species and location distributions.

Key Results

  • Performance Boost: Bias-handling strategies yielded a 1% to 5% gain in AUROC (Area Under the Receiver Operating Characteristic).
  • Model Comparison: Random Forests (RF) achieved an AUROC of 0.95 with bias handling, while Boosted Logistic Regression (BLR) reached 0.94.
  • Stability: Without bias handling, models performed at roughly 0.90 AUROC. The causally-informed models were more stable when applied to the external, shifted test set.
  • Efficiency: While RF performed slightly better, the authors noted that BLR is significantly faster to train and run, making it more suitable for real-time clinical or mobile sequencing applications.

Limitations and Future Work

  • Feature Bottleneck: The study only analyzed 20,000 $k$-mers out of 40+ million. The initial pre-selection of these 20,000 was based on "crude" (potentially biased) information gain, which might have filtered out important causal variants early on.
  • Single Realization: The external test set was limited to one country and one time period; more diverse "shifted" datasets are needed to prove generalizability.
  • Metagenomics: The authors suggest that these biases will be even more pronounced in metagenomics (samples containing multiple species, like gut or soil samples), which is a primary target for future research.
  • Methodological Gap: Traditional adjustment methods don't provide an advantage in every use case, suggesting that more specialized causal discovery algorithms for genomic data are needed.

Summary updated

Abstract

Highlights•Whole genome sequencing coupled with computational methods are increasingly used for predict antimicrobial resistance (AMR)•AMR databases and collections present sampling bias (geographical, temporal, and species-specific)•Causally-informed methods can aid the development of AMR prediction tools rendering them more robust with respect to bias
BibTeX
@article{prosperi2022assessing,
  author = {Mattia Prosperi and Christina Boucher and Jiang Bian and Simone Marini},
  title = {{Assessing putative bias in prediction of anti-microbial resistance from real-world genotyping data under explicit causal assumptions}},
  journal = {Artif. Intell. Medicine},
  year = {2022},
  doi = {10.1016/j.artmed.2022.102326},
}
Figure