Abstract
While recent multimodal large language models (MLLMs) have advanced automated ECG interpretation, they still face two key limitations: (1) insufficient multimodal synergy between time series signals and visual ECG representations, and (2) limited explainability in linking diagnoses to granular waveform evidence. We introduce GEM, the first MLLM unifying ECG time series, 12-lead ECG images, and text for grounded and clinician-aligned ECG interpretation. GEM enables feature-grounded analysis, evidence-driven reasoning, and a clinician-like diagnostic process through three core innovations: a dual-encoder framework extracting complementary time series and image features, cross-modal alignment for effective multimodal understanding, and knowledge-guided instruction generation for generating high-granularity grounding data (ECG-Grounding) linking diagnoses to measurable parameters (e.g., QRS/PR Intervals). Additionally, we propose the Grounded ECG Understanding task, a clinically motivated benchmark designed to comprehensively assess the MLLM's capability in grounded ECG understanding. Experimental results on both existing and our proposed benchmarks show GEM significantly improves predictive performance (CSN 7.4% ↑), explainability (22.7% ↑), and grounding (24.8% ↑), making it more suitable for real-world clinical applications.
Introduction
Automated Electrocardiogram (ECG) interpretation has the potential to revolutionize cardiac care, yet current models often operate as "black boxes." These systems typically focus on a single modality—either raw electrical signals (time series) or printed plots (images)—failing to capture the full breadth of information available to a human cardiologist. Furthermore, existing models lack "grounding," meaning they cannot explicitly link a diagnosis to the specific waveform features that justify it. To address these gaps, we present GEM, a Multimodal Large Language Model designed to provide transparent, evidence-based ECG interpretations by unifying time series, visual data, and clinical text.
Methodology
The Dual-Encoder Framework
GEM utilizes a dual-encoder architecture to process ECG data through two distinct lenses. A specialized ECG-CoCa encoder handles raw time series signals to capture fine-grained temporal dynamics, while a CLIP-based encoder processes 12-lead ECG images to identify global spatial structures. This synergy allows the model to benefit from the high-resolution data of the signal and the structural overview provided by the visual representation.
Cross-Modal Alignment
To integrate these disparate inputs, GEM employs a multi-stage projection system. Time series features are first aligned with the image feature space. Subsequently, both modalities are projected into a unified "language-like" embedding space. This allows the underlying Large Language Model (based on LLaVA or PULSE) to reason across both visual and signal-based evidence simultaneously.
Knowledge-Guided Instruction Generation
To overcome the scarcity of high-quality grounded data, we developed a pipeline to generate the ECG-Grounding dataset. This process involves:
- Feature Extraction: A "Grounding Feature Extractor" identifies 14 physiological parameters (e.g., QRS duration, ST-segment morphology) from raw signals.
- Diagnosis Guiding: These objective measurements are fed into a teacher model (GPT-4o) to generate 30,000 high-granularity instruction-response pairs. This ensures that every diagnosis in the training set is explicitly backed by measurable clinical evidence.
Evaluation and Benchmarking
We evaluated GEM using a combination of existing standards and a newly developed benchmark:
- ECG-Bench: Includes datasets such as PTB-XL, CPSC 2018, and CSN.
- Grounded ECG Understanding Task: A new benchmark specifically designed to measure a model's ability to identify diagnostic clues and provide supporting clinical knowledge.
- Human Validation: Eight board-certified cardiologists assessed GEM’s outputs for analytical relevance and clinical accuracy.
Key Results
GEM demonstrates significant improvements over state-of-the-art models like PULSE:
- Predictive Accuracy: Achieved a 7.4% increase in performance on the CSN dataset.
- Explainability and Grounding: Metrics for explainability and grounding improved by 22.7% and 24.8%, respectively.
- Lead Assessment: In the Grounded ECG Understanding task, GEM increased lead assessment coverage from 7.11% (PULSE) to over 71%.
- Clinical Alignment: Human experts rated GEM’s outputs highly (averaging >4/5), noting its ability to mimic a clinician's diagnostic reasoning process.
Discussion and Future Work
GEM represents a significant step toward transparent AI in cardiology. By learning fundamental cardiac principles rather than simple pattern matching, the model showed strong generalization capabilities on out-of-domain datasets like PTB-XL.
However, limitations remain. The model occasionally struggles with very subtle morphological changes, such as minor ST-segment deviations, and may overstate condition severity due to a lack of broader patient clinical history. Future research will focus on integrating "human-in-the-loop" feedback and incorporating longitudinal patient data to further refine GEM’s clinical judgment.
Summary updated