Generalization Challenges in ECG Deep Learning: Insights from Dataset Characteristics and Attention Mechanism

Zhaojing Huang, Sarisha MacLachlan, Leping Yu, Luis Fernando Herbozo Contreras, Nhan Duy Truong, Antônio H. Ribeiro, Omid KaveheimedRxiv 2023 · DOI

Abstract

This research investigates the influence of dataset characteristics on the performance and generalization capabilities of deep learning models on ECG data. The study evaluates multiple subsets of the TNMG dataset with varying levels of curated characteristics to assess their impact on model performance. Additionally, an attention mechanism is introduced to enhance model accuracy and generalization. The experimental results reveal that models trained on balanced subsets and incorporating the attention mechanism consistently outperform those trained on unbalanced data or without attention, emphasizing the critical importance of dataset balance and attention mechanism for achieving improved model performance. Surprisingly, the largest ECG dataset, TNMG, proved less effective in generalization than smaller, curated subsets. The study demonstrates that a well-balanced and thoughtfully curated dataset, combined with the attention mechanism, can lead to competitive model performance, even with a significantly smaller size.

Introduction

Electrocardiograms (ECGs) are the gold standard for diagnosing heart arrhythmias, yet their interpretation is labor-intensive and requires high-level expertise. While Deep Learning (DL) models have shown remarkable performance in controlled studies, they often suffer from a lack of generalizability when deployed in real-world clinical settings. This "generalization gap" is often caused by dataset shift—differences in equipment, patient demographics, and disease prevalence between training and deployment sites. This research explores how architectural enhancements, specifically attention mechanisms, and strategic data curation can bridge this gap.

Methodology: Architecture and Attention

The researchers utilized a high-performing base model consisting of a 1D Residual Neural Network (ResNet). This architecture is well-suited for time-series data like ECGs, using convolutional layers and residual blocks to learn deep features without signal degradation.

The primary innovation in this study is the addition of a Bahdanau-style attention mechanism placed before the final classification step. In the context of an ECG, the attention mechanism acts as a highlighter, calculating alignment scores to determine which segments of the 10-second heart rhythm are most relevant to a specific diagnosis (e.g., focusing on the PR interval for heart blocks).

Schematic of the 1D ResNet architecture and the detailed components of a residual block.

Dataset Curation and Characteristics

To test the impact of data quality, the study used the Telehealth Network Minas Gerais (TNMG) dataset from Brazil, containing 2.3 million recordings. The researchers created four distinct training subsets (I-IV), each with 21,000 traces but different selection criteria:

  • Subset I: Perfectly balanced (3,000 of each arrhythmia type + 3,000 normal).
  • Subset II: Favors "complex" patients with multiple arrhythmias.
  • Subset III: Age-balanced across the lifespan.
  • Subset IV: Randomly selected (control group).

External validation was performed using the CPSC (China Physiological Signal Challenge) and SNH (Shaoxing and Ningbo Hospitals) datasets to assess cross-continental generalization.

Visual representation of dataset characteristics, including age/gender distributions and arrhythmia prevalence across subsets.

Key Results and Insights

Bigger Isn't Always Better

A major finding was that models trained on smaller, curated subsets (specifically Subset III) often generalized better than models trained on the entire multi-million-record TNMG dataset. This challenges the common assumption that more data is always the answer in deep learning.

Attention as a Generalizer

The attention mechanism improved accuracy and acted as a buffer against noise. By forcing the model to focus on clinically relevant features, it became less likely to overfit to site-specific recording artifacts.

  • Internal Performance: On the TNMG test set, the attention-boosted model achieved an F1 score of 0.875, compared to 0.844 for the base model.
  • Cross-Dataset Success: On the CPSC dataset, the attention model reached an F1 of 0.879, outperforming the original base model (0.867).

The Necessity of Balance

Subset IV (randomly selected) performed the worst across all tests. This highlights that in medical AI, representing rare conditions is more important than matching the natural, healthy-dominant distribution of the general population.

Conclusion

This research underscores the critical importance of dataset curation and attention mechanisms in biomedical machine learning. While the performance drop on the highly skewed SNH dataset (F1 ~0.52) shows that dataset shift remains a formidable challenge, the study proves that well-balanced, thoughtfully curated datasets can outperform massive, unbalanced ones. Future work will explore more sophisticated architectures like Transformers and self-supervised learning to further improve clinical robustness.

Summary updated

Abstract

Abstract This research investigates the influence of dataset characteristics on the performance and generalization capabilities of deep learning models, on ECG data. The study evaluates multiple subsets of the TNMG dataset with varying levels of curated characteristics to assess their impact on model performance. Additionally, an attention mechanism is introduced to enhance model accuracy and generalization. The experimental results reveal that models trained on balanced subsets and incorporating the attention mechanism consistently outperform those trained on unbalanced data or without attention, emphasizing the critical importance of dataset balance and attention mechanism for achieving improved model performance. Surprisingly, the largest ECG dataset, TNMG, proved less effective in generalization than smaller, curated subsets. The study demonstrates that a well-balanced and thoughtfully curated dataset, combined with the attention mechanism, can lead to competitive model performance, even with a significantly smaller size. This research on ECG data underscores the critical importance of dataset curation, balance, and attention mechanisms in biomedical machine learning. It highlights that well-balanced, thoughtfully curated datasets with attention mechanisms can outperform larger, unbalanced datasets, challenging conventional notions and offering potential advancements in medical data analysis and patient care.
BibTeX
@misc{huang2023generalization,
  author = {Zhaojing Huang and Sarisha MacLachlan and Leping Yu and Luis Fernando Herbozo Contreras and Nhan Duy Truong and Antônio H. Ribeiro and Omid Kavehei},
  title = {{Generalization Challenges in ECG Deep Learning: Insights from Dataset Characteristics and Attention Mechanism}},
  year = {2023},
  doi = {10.1101/2023.07.05.23292238},
  howpublished = {medRxiv},
}
Figure