Abstract
Predicting antimicrobial resistance (AMR) from genomic data is often compromised by sampling biases inherent in large public databases. These databases are frequently influenced by specific outbreaks, geographic regions, or temporal trends, leading to models that learn spurious correlations rather than true causal relationships. This study investigates the application of causal inference techniques—such as propensity score matching and covariate adjustment—to improve the robustness of machine learning models. By accounting for confounders like bacterial species, geographic location, and collection year, we demonstrate that "causally-informed" models achieve superior performance (AUROC up to 0.95) and maintain stability when faced with real-world distribution shifts.
Introduction
The use of Whole Genome Sequencing (WGS) to predict AMR phenotypes is becoming a cornerstone of modern microbiology. However, the training data for these models, often sourced from repositories like the Pathosystems Resource Integration Center (PATRIC), is not a random sample of the global bacterial population. Instead, it reflects the specific interests and resources of contributing laboratories.
This non-random sampling creates a significant distribution shift. A predictive model might identify a genetic signature as a strong predictor of resistance simply because that signature is prevalent in a species that was oversampled in a particular year or country. Consequently, when such models are deployed on new data from different contexts, their predictive accuracy often declines. This research aims to bridge the gap between "black-box" machine learning and causal understanding to develop more reliable AMR prediction tools.
Methods
The researchers framed the presence of specific genetic traits (represented as $k$-mers) as an "exposure" and the resulting AMR phenotype as the "outcome." To address potential biases, they employed several causal inference strategies:
- Causal Modeling: A Partially Directed Acyclic Graph (PDAG) was constructed to visualize and formalize the assumed relationships between metadata (Year, Country), Species, Genetic Signatures, and AMR (see Figure 1).
- Propensity Score Matching (PSM): This technique was used to rebalance the dataset, creating a synthetic environment where the probability of a genetic trait appearing is balanced across different confounding factors.
- Covariate Adjustment: Models were tested that explicitly included species, country, and year as features alongside genomic data.
- ATE-based Feature Selection: Genetic signatures were ranked based on their Average Treatment Effect (ATE) after adjusting for confounders, rather than simple correlation.
The study utilized a training set of 9,851 bacterial instances from PATRIC, focusing on tetracycline resistance. An external test set of 1,085 instances from a specific geographic and temporal slice (USA, most recent quartile) was used to simulate real-world distribution shifts. Two primary architectures were evaluated: Boosted Logistic Regression (BLR) and Random Forests (RF).
Results
The integration of causal bias-handling strategies led to measurable improvements in model performance and stability.
- Performance Gains: Causally-informed models showed a 1% to 5% increase in AUROC. Random Forests achieved the highest AUROC of 0.95, while Boosted Logistic Regression reached 0.94.
- Robustness to Distribution Shift: The most significant advantage of causal adjustment was observed in the external test set. While standard models often saw performance drops, the causally-informed versions maintained high sensitivity and specificity (Figure 2).
- Impact on Genetic Signatures: Adjusting for bias changed the perceived direction of effect for 8.1% of $k$-mers and removed statistical significance for 20.2% of them, effectively filtering out "noise" generated by confounding factors.
Discussion and Conclusion
The study demonstrates that genetic signatures often act as proxy confounders for bacterial species. While these proxies can aid prediction in static environments, they fail under distribution shifts. Causal adjustment provides a path toward more generalizable models.
However, limitations remain. The initial selection of 20,000 $k$-mers from a pool of 40 million was based on potentially biased information gain, which may have excluded relevant causal variants. Future work should explore more diverse "shifted" datasets and apply these methods to metagenomics, where species-level confounding is even more complex. While Random Forests performed slightly better, the computational efficiency of Boosted Logistic Regression makes it a strong candidate for real-time clinical applications. Ultimately, moving beyond simple correlation toward causal understanding is essential for the reliable deployment of genomic AMR prediction.
Summary updated