Domain shifts decrease the accuracy of machine learning algorithms to detect myocardial ischemia in ECG recordings: A multi-database analysis

Sandra Frank, Martin W. Dünser, Thomas Tschoellitsch, Ákos Filakovszky, Victoria Habsburg-Lothringen, Günter Klambauer, Jens MeierResearch Square 2025 · DOI

Introduction

Machine learning (ML) algorithms have demonstrated exceptional accuracy in detecting myocardial ischemia from electrocardiograms (ECGs) in research settings. However, their transition to clinical practice is hindered by uncertainty regarding their reliability across different healthcare environments. This study investigates the impact of "domain shifts"—variations in equipment, patient demographics, and data labeling—on the performance of these algorithms.

Methods

The researchers utilized a standardized pipeline to analyze 55,953 12-lead ECG recordings from six major public databases: CPSC 2018, CPSC 2018 extra, PTB, PTB-XL, GEO, and STP.

Signal Processing and Modeling

ECG signals were filtered to remove baseline wander, and 500ms "sweeps" around the R-spike were extracted to create representative beats. Six distinct ML approaches were evaluated:

  • Generalized Linear Models (GLM)
  • Random Forests (RF)
  • Gradient Boosting Machines (GBM)
  • XGBoost
  • Deep Neural Networks (DNN)
  • Ensemble Learning (AML)

Experimental Scenarios

To assess generalization, models were tested under three conditions:

  1. Single: Trained and tested on the same dataset (the "lab" scenario).
  2. All: Trained on a pooled dataset from all sources.
  3. Leave-one-dataset-out: Trained on five datasets and tested on an unseen sixth dataset to simulate real-world deployment.
Figure 1: Overview of the study workflow, including data sources, signal preprocessing, machine learning models, and evaluation scenarios.

Key Insights

The "Site ID" Phenomenon

A significant finding was that models could accurately identify the source database of a recording based on the ECG signal alone. This suggests that datasets contain "fingerprints" related to local hardware or protocols rather than purely physiological data. If an algorithm learns the "noise" of one database rather than universal biological signals, it becomes a brittle tool that cannot generalize.

Feature Selection Shifts

Using the Boruta algorithm for feature selection, the study found that models trained on single sites often relied on arbitrary time points in the ECG. Conversely, when forced to learn from pooled data, the models shifted their focus to the ST-segment and T-wave—the same features used by human cardiologists to diagnose ischemia.

Results

When trained and tested within the same distribution, models achieved high performance with Area Under the Curve (AUC) values between 0.913 and 0.938. However, performance declined notably in the "leave-one-out" validation. For instance, the best ensemble model reached an AUC of only 0.854 when tested on the PTB-XL database after being trained on others.

Most critically, sensitivity (the ability to detect actual ischemia) dropped significantly in cross-dataset tests, sometimes falling to ≤ 0.039. This means the models frequently failed to flag the very pathology they were designed to find when faced with unfamiliar data distributions, even though specificity remained relatively stable.

Conclusion

While ML algorithms are highly effective within familiar data distributions, their ability to generalize across diverse clinical settings is limited by domain shifts. Simply increasing the volume of data is insufficient; future research must focus on domain adaptation—teaching AI to ignore site-specific noise in favor of universal physiological markers. Until then, these models require rigorous external validation before they can be trusted at the bedside.

Summary updated

Abstract

<title>Abstract</title> Machine learning algorithms have shown excellent accuracy in detecting myocardial ischemia from electrocardiograms, but their clinical reliability remains uncertain. In this study, we explored how domain shifts between different datasets affect the performance of these algorithms. We used six publicly available 12-lead ECG databases, containing a total of 55,953 recordings, and applied several machine learning approaches, including generalized linear models, random forests, gradient boosting, deep neural networks, and ensemble learning. The models were evaluated under different conditions—within individual datasets, across all datasets combined, and using leave-one-dataset-out validation.When trained and tested on the same dataset, the models performed very well, with area under the curve values exceeding 0.90. However, performance dropped notably when the models were tested on data from different sources. Sensitivity decreased substantially, while specificity remained relatively stable. Further analysis showed that variations in ECG patterns across datasets contributed to these differences in performance.Overall, our results demonstrate that while machine learning algorithms can detect myocardial ischemia accurately within familiar data, their ability to generalize across diverse datasets is limited. Addressing dataset heterogeneity and improving model robustness will be essential before such systems can be reliably implemented in clinical practice.
BibTeX
@inproceedings{frank2025domain,
  author = {Sandra Frank and Martin W. Dünser and Thomas Tschoellitsch and Ákos Filakovszky and Victoria Habsburg-Lothringen and Günter Klambauer and Jens Meier},
  title = {{Domain shifts decrease the accuracy of machine learning algorithms to detect myocardial ischemia in ECG recordings: A multi-database analysis}},
  booktitle = {Research Square},
  year = {2025},
  doi = {10.21203/rs.3.rs-7893141/v1},
}
Figure