Introduction
Foundation models—large-scale neural networks pretrained on vast, unlabeled datasets—are increasingly being applied to medical domains, including electrocardiography (ECG). While these models offer the promise of high-performance clinical decision support, their internal representations often remain a "black box." A critical concern is how the composition of pretraining data, such as patient demographics and health status, influences a model's final accuracy and its ability to generalize to "out-of-distribution" (OOD) data from different hospital systems or geographic regions.
In this study, the authors investigate a specific pitfall in contrastive learning: the tendency of models to learn hardware-specific artifacts rather than underlying cardiac biology. They introduce a novel training strategy, In-Distribution Batching (IDB), to mitigate this issue and enhance the robustness of ECG foundation models.
Methodology: The CAPE Model and IDB
The authors developed the CAPE (Contrasting by Augmented Patient Electrocardiograms) foundation model using a four-layer ResNet backbone. The model employs a contrastive learning framework based on the InfoNCE loss function. In this setup, the model is tasked with mapping two different ECG segments from the same patient to similar mathematical representations (embeddings) while pushing embeddings from different patients apart.
The Challenge of Hardware Artifacts
When a model is trained on data aggregated from multiple hospitals, it often learns to distinguish between sites based on subtle differences in ECG machine hardware or signal processing techniques. This "technical noise" allows the model to achieve high accuracy on training data by identifying the source hospital rather than the patient's clinical state. Consequently, such models often fail when deployed in new clinical environments with different hardware.
In-Distribution Batching (IDB)
To address this, the authors proposed In-Distribution Batching (IDB). In this approach, each training batch is restricted to a single cohort or hospital. By ensuring that "negative samples" (the patients the model must distinguish between) come from the same site, the model is forced to ignore site-wide hardware signatures. This "nulls out" technical artifacts and compels the model to focus on intrinsic clinical features, leading to the CAPE-Z variant.
Data Diversity and Experimental Setup
The study leveraged a massive dataset of over 5.2 million ECGs spanning three continents:
- Pretraining Cohorts (BCSV): Included secondary care settings with high disease prevalence (BIDMC & VUMC, USA) and primary care settings (CODE, Brazil; SHZS, China).
- Downstream Evaluation: The models were evaluated on age regression and sex classification tasks using frozen features to isolate the quality of the learned representations. Evaluation sets included the UK Biobank (healthy volunteers) and PTB-XL (secondary care, Germany).
Key Insights
The Value of "Sicker" Data
A significant finding was that models pretrained on secondary care cohorts (BIDMC, VUMC) outperformed those trained on healthier populations. The authors suggest that sicker patients provide a "richer" textbook for the AI, as their ECGs exhibit a wider and more diverse range of cardiac signal patterns compared to healthy volunteers.
Visualizing the Hardware Signature
Using t-SNE visualizations, the authors confirmed that standard multi-center training caused the model to cluster data by recording device. This clustering disappeared when IDB was implemented, confirming that the model was no longer relying on technical noise to differentiate samples.
The UK Biobank "Mirage"
The study also highlighted the risks of narrow data distributions. While models performed exceptionally well on the UK Biobank, this was largely attributed to the dataset's narrow age range (mostly 64–66 years). This serves as a cautionary tale: high performance on specific benchmarks may reflect data quirks rather than true general intelligence.
Results and Generalization
The impact of IDB on OOD generalization was profound:
- Standard Multi-center Training (CAPE-X): While achieving strong results on in-distribution data (Age MAE: 7.86; Sex AUROC: 0.955), performance collapsed on new cohorts. For instance, the Age Mean Absolute Error (MAE) on the Brazilian CODE cohort was a staggering 24.0 years.
- IDB Strategy (CAPE-Z): Implementing IDB slashed the Age MAE on the Brazilian cohort from 24.0 down to 7.96 years. This statistically significant improvement demonstrates that CAPE-Z is far more clinically relevant across different healthcare systems.
Conclusion and Future Work
This research underscores that what a foundation model learns is heavily dictated by how the data is batched during pretraining. By preventing models from "cheating" via hardware artifacts, IDB significantly improves the generalisability of ECG representations.
Future work will involve fine-tuning the model backbones rather than using frozen features, expanding the evaluation to include specific disease diagnoses like arrhythmias, and further validating the clinical fairness of these models across diverse demographic subgroups.
Summary updated