TL;DR
GEM is the first multimodal large language model (MLLM) that unifies ECG time series data, 12-lead ECG images, and text to achieve grounded, clinician-aligned ECG interpretation. By combining a dual-encoder architecture with a novel knowledge-guided data generation pipeline, GEM significantly improves diagnostic accuracy and explicitly links its conclusions to granular waveform evidence.
What Problem Does This Paper Address?
Automated electrocardiogram (ECG) interpretation has advanced rapidly with the rise of Multimodal Large Language Models (MLLMs). However, current state-of-the-art models suffer from two major limitations that prevent them from being fully trusted in real-world clinical settings:
- Limited Modality Exploration: Clinicians naturally integrate both dynamic, machine-measured temporal signals and spatial waveform patterns from 12-lead printed plots. In contrast, existing AI models typically process only one non-text modality—either the raw time series or a static image—missing out on the complementary strengths of both.
- Insufficient Explainability and Grounding: Most models act as "black boxes," outputting a diagnosis without explaining why. A trustworthy model needs to provide "feature-grounded" analysis, explicitly linking a diagnosis (e.g., right bundle branch block) to specific, measurable physiological parameters (e.g., prolonged QRS duration or specific wave patterns).
The authors introduce GEM to bridge these gaps, simulating a realistic, evidence-driven diagnostic process akin to a human cardiologist.
Methods
GEM's architecture consists of three core innovations designed to handle multimodal inputs and enforce clinical explainability:
- Multimodal Encoding (Dual-Encoder Framework): To capture both local signal dynamics and global visual structures, GEM uses two separate, pre-trained encoders. It adopts
ECG-CoCafor the raw ECG time series and aCLIPvision encoder (from LLaVA) for the transformed 12-lead ECG images. - Cross-Modal Alignment Learning: Because the time series and vision encoders are trained independently, their representation spaces are inconsistent. GEM maps the time series features to match the dimensionality of the image features using a Multi-Layer Perceptron (MLP), and then utilizes a shared projector to translate both modalities into language-like embeddings that the LLM can comprehend.
- Knowledge-Guided Instruction Data Generation: To train the model to "show its work," the authors built a pipeline that extracts 14 physiological feature sequences (such as Heart Rate, PR Interval, QRS Duration, and ST Form) across all 12 leads using a deterministic, rule-based tool (
FeatureDB). A cardiology-specificDiagnosis Guiderstructures these measurements into prompts for GPT-4o, which then curates a highly detailed, grounded dataset called ECG-Grounding.
Datasets
The model was trained and evaluated using a mix of massive existing datasets and a newly proposed benchmark:
- Training Data:
ECG-Instruct(from PULSE): Contains 1,156,110 standard ECG-text conversations.ECG-Grounding(New): 30,000 fine-grained instruction-response pairs generated from theMIMIC-IV-ECGdatabase, annotated with heartbeat-level physiological features.- Evaluation Benchmarks:
Grounded ECG Understanding Task(New): A clinically motivated benchmark designed to test a model's ability to identify diagnostic clues across multiple leads. Evaluated on test sets fromMIMIC-IV-ECG(in-domain) andPTB-XL(out-domain).ECG-Bench: An existing benchmark used to evaluate cardiac abnormality detection and report generation across diverse datasets (PTB-XL,CPSC2018,CODE-15%,CSN, andG12EC).
Key Insights and Ideas
- The Power of "Time Series + Image": Time series models excel at capturing dynamic, heartbeat-to-heartbeat changes but can miss global spatial structures. Vision models see the big picture but overlook subtle temporal details. Fusing them creates a synergistic effect that mirrors a clinician's actual workflow.
- Bootstrapping Grounding without Human Costs: High-granularity clinical data annotation is incredibly expensive. By using deterministic signal processing algorithms to extract exact wave measurements first, the authors successfully guided a frontier LLM (GPT-4o) to generate highly accurate, complex medical training data without manual expert labeling.
- One-Step Training Efficiency: Unlike standard MLLM pipelines that use a multi-stage training process (aligning projectors first, then fine-tuning the LLM), GEM uses a one-step approach. Freezing the encoders while jointly training the projectors and the LLM enhanced cross-modal consistency, even with limited data.
Key Results
GEM demonstrated substantial improvements over state-of-the-art models like PULSE across all primary evaluation metrics.
Grounded ECG Understanding
On the newly introduced Grounded ECG Understanding task, GEM achieved massive leaps in explainability and lead coverage:
- Diagnosis Accuracy: Reached 87.24% on MIMIC-IV-ECG (outperforming PULSE's 81.14%) and 73.59% on the out-domain PTB-XL dataset (outperforming PULSE's 59.24%), proving strong generalization capabilities.
- Lead Assessment Coverage: Jumped from a meager ~7-11% (PULSE) to over 71-79% (GEM), showing that GEM systematically evaluates almost all 12 leads rather than fixating on a few.
- ECG Feature Grounding: Improved by roughly 25%, scoring around 75 out of 100, meaning the majority of GEM's diagnostic conclusions were explicitly linked to measurable ECG parameters.
Abnormality Detection
On standard classification tasks within ECG-Bench, GEM consistently achieved top-tier performance, notably hitting an accuracy of 92.6% on the CSN dataset.
Limitations and Future Work
Despite its success, the authors note several limitations that must be addressed before clinical deployment:
- Encoder Deficiencies: Subtle morphological patterns, such as minor ST-segment changes or P-wave abnormalities, were occasionally missed by the frozen encoders during the representation stage, leading to incorrect diagnoses.
- Overstating Severity: GEM sometimes exaggerated the severity of certain conditions. Because it interprets ECGs in isolation without access to a patient's broader clinical history or context, it can lack the nuance a physician naturally applies.
- Data Generation Alignment: While GPT-4o generated high-quality training data, it still occasionally produced target answers that misaligned with expert cardiologist interpretations.
Future Directions: The team plans to explore incorporating expert human feedback directly into the data generation loop (e.g., RLHF) and integrating patient clinical history to refine the model's safety and clinical alignment.
Summary updated