Introduction
Machine learning (ML) algorithms have demonstrated exceptional accuracy in detecting myocardial ischemia from electrocardiograms (ECGs) in research settings. However, their transition to clinical practice is hindered by uncertainty regarding their reliability across different healthcare environments. This study investigates the impact of "domain shifts"—variations in equipment, patient demographics, and data labeling—on the performance of these algorithms.
Methods
The researchers utilized a standardized pipeline to analyze 55,953 12-lead ECG recordings from six major public databases: CPSC 2018, CPSC 2018 extra, PTB, PTB-XL, GEO, and STP.
Signal Processing and Modeling
ECG signals were filtered to remove baseline wander, and 500ms "sweeps" around the R-spike were extracted to create representative beats. Six distinct ML approaches were evaluated:
- Generalized Linear Models (GLM)
- Random Forests (RF)
- Gradient Boosting Machines (GBM)
- XGBoost
- Deep Neural Networks (DNN)
- Ensemble Learning (AML)
Experimental Scenarios
To assess generalization, models were tested under three conditions:
- Single: Trained and tested on the same dataset (the "lab" scenario).
- All: Trained on a pooled dataset from all sources.
- Leave-one-dataset-out: Trained on five datasets and tested on an unseen sixth dataset to simulate real-world deployment.
Key Insights
The "Site ID" Phenomenon
A significant finding was that models could accurately identify the source database of a recording based on the ECG signal alone. This suggests that datasets contain "fingerprints" related to local hardware or protocols rather than purely physiological data. If an algorithm learns the "noise" of one database rather than universal biological signals, it becomes a brittle tool that cannot generalize.
Feature Selection Shifts
Using the Boruta algorithm for feature selection, the study found that models trained on single sites often relied on arbitrary time points in the ECG. Conversely, when forced to learn from pooled data, the models shifted their focus to the ST-segment and T-wave—the same features used by human cardiologists to diagnose ischemia.
Results
When trained and tested within the same distribution, models achieved high performance with Area Under the Curve (AUC) values between 0.913 and 0.938. However, performance declined notably in the "leave-one-out" validation. For instance, the best ensemble model reached an AUC of only 0.854 when tested on the PTB-XL database after being trained on others.
Most critically, sensitivity (the ability to detect actual ischemia) dropped significantly in cross-dataset tests, sometimes falling to ≤ 0.039. This means the models frequently failed to flag the very pathology they were designed to find when faced with unfamiliar data distributions, even though specificity remained relatively stable.
Conclusion
While ML algorithms are highly effective within familiar data distributions, their ability to generalize across diverse clinical settings is limited by domain shifts. Simply increasing the volume of data is insufficient; future research must focus on domain adaptation—teaching AI to ignore site-specific noise in favor of universal physiological markers. Until then, these models require rigorous external validation before they can be trusted at the bedside.
Summary updated