Abstract
This research investigates the influence of dataset characteristics on the performance and generalization capabilities of deep learning models on ECG data. The study evaluates multiple subsets of the TNMG dataset with varying levels of curated characteristics to assess their impact on model performance. Additionally, an attention mechanism is introduced to enhance model accuracy and generalization. The experimental results reveal that models trained on balanced subsets and incorporating the attention mechanism consistently outperform those trained on unbalanced data or without attention, emphasizing the critical importance of dataset balance and attention mechanism for achieving improved model performance. Surprisingly, the largest ECG dataset, TNMG, proved less effective in generalization than smaller, curated subsets. The study demonstrates that a well-balanced and thoughtfully curated dataset, combined with the attention mechanism, can lead to competitive model performance, even with a significantly smaller size.
Introduction
Electrocardiograms (ECGs) are the gold standard for diagnosing heart arrhythmias, yet their interpretation is labor-intensive and requires high-level expertise. While Deep Learning (DL) models have shown remarkable performance in controlled studies, they often suffer from a lack of generalizability when deployed in real-world clinical settings. This "generalization gap" is often caused by dataset shift—differences in equipment, patient demographics, and disease prevalence between training and deployment sites. This research explores how architectural enhancements, specifically attention mechanisms, and strategic data curation can bridge this gap.
Methodology: Architecture and Attention
The researchers utilized a high-performing base model consisting of a 1D Residual Neural Network (ResNet). This architecture is well-suited for time-series data like ECGs, using convolutional layers and residual blocks to learn deep features without signal degradation.
The primary innovation in this study is the addition of a Bahdanau-style attention mechanism placed before the final classification step. In the context of an ECG, the attention mechanism acts as a highlighter, calculating alignment scores to determine which segments of the 10-second heart rhythm are most relevant to a specific diagnosis (e.g., focusing on the PR interval for heart blocks).
Dataset Curation and Characteristics
To test the impact of data quality, the study used the Telehealth Network Minas Gerais (TNMG) dataset from Brazil, containing 2.3 million recordings. The researchers created four distinct training subsets (I-IV), each with 21,000 traces but different selection criteria:
- Subset I: Perfectly balanced (3,000 of each arrhythmia type + 3,000 normal).
- Subset II: Favors "complex" patients with multiple arrhythmias.
- Subset III: Age-balanced across the lifespan.
- Subset IV: Randomly selected (control group).
External validation was performed using the CPSC (China Physiological Signal Challenge) and SNH (Shaoxing and Ningbo Hospitals) datasets to assess cross-continental generalization.
Key Results and Insights
Bigger Isn't Always Better
A major finding was that models trained on smaller, curated subsets (specifically Subset III) often generalized better than models trained on the entire multi-million-record TNMG dataset. This challenges the common assumption that more data is always the answer in deep learning.
Attention as a Generalizer
The attention mechanism improved accuracy and acted as a buffer against noise. By forcing the model to focus on clinically relevant features, it became less likely to overfit to site-specific recording artifacts.
- Internal Performance: On the TNMG test set, the attention-boosted model achieved an F1 score of 0.875, compared to 0.844 for the base model.
- Cross-Dataset Success: On the CPSC dataset, the attention model reached an F1 of 0.879, outperforming the original base model (0.867).
The Necessity of Balance
Subset IV (randomly selected) performed the worst across all tests. This highlights that in medical AI, representing rare conditions is more important than matching the natural, healthy-dominant distribution of the general population.
Conclusion
This research underscores the critical importance of dataset curation and attention mechanisms in biomedical machine learning. While the performance drop on the highly skewed SNH dataset (F1 ~0.52) shows that dataset shift remains a formidable challenge, the study proves that well-balanced, thoughtfully curated datasets can outperform massive, unbalanced ones. Future work will explore more sophisticated architectures like Transformers and self-supervised learning to further improve clinical robustness.
Summary updated