Abstract
Machine Learning (ML) models have gained popularity in medical imaging analysis given their expert level performance in many medical domains. To enhance the trustworthiness, acceptance, and regulatory compliance of medical imaging models and to facilitate their integration into clinical settings, we review and categorise methods for ensuring ML reliability, both during development and throughout the model's lifespan. Specifically, we provide an overview of methods assessing models' inner-workings regarding bias encoding and detection of data drift for disease classification models. Additionally, to evaluate the severity in case of a significant drift, we provide an overview of the methods developed for classifier accuracy estimation in case of no access to ground truth labels. This should enable practitioners to implement methods ensuring reliable ML deployment and consistent prediction performance over time.
Introduction: The Deployment Gap in Medical AI
While Machine Learning (ML) models often achieve expert-level performance in controlled laboratory settings, their transition to real-world clinical environments is frequently hindered by a "deployment gap." This gap is primarily driven by two "silent killers" of model performance: hidden biases and data drift.
Bias encoding occurs when models inadvertently learn "shortcuts"—relying on superficial features like scanner artifacts, hospital-specific protocols, or demographic markers rather than actual pathology. Data drift, on the other hand, arises because the real world is non-stationary. Changes in patient demographics, the emergence of new diseases, or upgrades to imaging hardware can cause "live" data to diverge from training data, leading to silent performance degradation.
To address these risks, a systematic approach is required to assess models throughout their lifecycle, from pre-deployment bias checks to post-deployment drift monitoring and re-evaluation.
Methodological Pillars for Reliable ML
The survey categorizes methods into three main pillars designed to ensure reliability and trustworthiness.
1. Bias Encoding Assessment
Instead of merely observing output disparities, these methods probe a model's internal representations. A key technique involves using "frozen backbones" to determine if sensitive attributes, such as race or sex, are implicitly learned by the model. Furthermore, causal inference is applied to distinguish between genuine biological correlations and spurious confounding factors.
2. Data Drift Detection
Detecting distribution shifts in high-dimensional medical image data is critical. The survey reviews several techniques:
- Blackbox Shift Detection (BBSD): Monitors the model's softmax outputs to identify changes in prediction patterns.
- Autoencoders (UAE/TAE): Both untrained and trained autoencoders can detect shifts by measuring how well a model reconstructs new images compared to the training set.
3. Label-Free Performance Estimation
In clinical practice, expert ground-truth labels may not be immediately available. Methods like Automatic Model Evaluation (AutoEval) and Weighted Flips (WF) use distribution distances and prediction stability to estimate the current error rate without needing immediate human verification.
Datasets and Benchmarking
The survey references a diverse array of datasets used to validate these reliability methods. While standard baselines like MNIST and CIFAR-10 serve for initial sanity checks, clinical validation relies on large-scale medical benchmarks such as CheXpert (chest X-rays), MURA (musculoskeletal radiographs), and MIMIC-CXR. Additionally, the SimBA framework is highlighted for its ability to generate synthetic neuroimaging data with controlled, "known" biases, providing a sandbox for testing detection algorithms.
Key Insights and Results
A critical insight from recent research is the distinction between encoding and usage. A model might encode sensitive information (like patient race) in its latent space without necessarily using it for diagnosis. Distinguishing these is vital for fairness. Moreover, models are "lazy" learners; for instance, if a specific portable X-ray machine is predominantly used in the ICU, a model might learn to identify the machine's signature as a proxy for disease.
Synthesis of findings across the literature suggests:
- TAE + BBSD (combining autoencoders with blackbox detection) is consistently the most sensitive method for detecting subtle, malignant shifts.
- Reverse Classification Accuracy (RCA) is a leading method for estimating segmentation quality without human re-labeling.
- Simple confidence scores (entropy) are often ineffective for drift detection, as models can be "confidently wrong" on out-of-distribution data.
Limitations and Future Directions
Despite progress, several challenges remain. Most current methods are validated on simulated shifts (e.g., added noise) rather than "natural" drift in long-term deployments. There is also a need for better Root Cause Analysis to determine why a shift occurred—whether due to hardware updates or population changes. Finally, as the field moves toward large-scale "foundation models," understanding how biases from massive pre-training data trickle down to clinical tasks remains a critical area for future research.
Summary updated