GEM: Empowering MLLM for Grounded ECG Understanding with Time Series and Images

Xiang Lan, Feng Wu, Kai He, Qinghao Zhao, Shenda Hong, Mengling Feng

TL;DR

GEM is the first multimodal large language model (MLLM) that unifies ECG time series data, 12-lead ECG images, and text to achieve grounded, clinician-aligned ECG interpretation. By combining a dual-encoder architecture with a novel knowledge-guided data generation pipeline, GEM significantly improves diagnostic accuracy and explicitly links its conclusions to granular waveform evidence.

What Problem Does This Paper Address?

Automated electrocardiogram (ECG) interpretation has advanced rapidly with the rise of Multimodal Large Language Models (MLLMs). However, current state-of-the-art models suffer from two major limitations that prevent them from being fully trusted in real-world clinical settings:

  1. Limited Modality Exploration: Clinicians naturally integrate both dynamic, machine-measured temporal signals and spatial waveform patterns from 12-lead printed plots. In contrast, existing AI models typically process only one non-text modality—either the raw time series or a static image—missing out on the complementary strengths of both.
  2. Insufficient Explainability and Grounding: Most models act as "black boxes," outputting a diagnosis without explaining why. A trustworthy model needs to provide "feature-grounded" analysis, explicitly linking a diagnosis (e.g., right bundle branch block) to specific, measurable physiological parameters (e.g., prolonged QRS duration or specific wave patterns).

The authors introduce GEM to bridge these gaps, simulating a realistic, evidence-driven diagnostic process akin to a human cardiologist.

Methods

GEM's architecture consists of three core innovations designed to handle multimodal inputs and enforce clinical explainability:

Figure 2: GEM’s Architecture. Multimodal Encoding: Separate encoders process ECG time series and images to generate modality-specific representations, enabling a holistic analysis of ECG data. Cross-modal Alignment Learning: Time series and image representations are first aligned and then mapped to a textual space using a shared projector, ensuring coherent understanding for the LLM. Knowledge-guided Instruction Data Generation: Physiological features extracted from all 12 leads are sequenced and structured using a diagnosis guider, which prompts GPT-4o with domain-specific instructions to generate high-granularity instructional data.
  • Multimodal Encoding (Dual-Encoder Framework): To capture both local signal dynamics and global visual structures, GEM uses two separate, pre-trained encoders. It adopts ECG-CoCa for the raw ECG time series and a CLIP vision encoder (from LLaVA) for the transformed 12-lead ECG images.
  • Cross-Modal Alignment Learning: Because the time series and vision encoders are trained independently, their representation spaces are inconsistent. GEM maps the time series features to match the dimensionality of the image features using a Multi-Layer Perceptron (MLP), and then utilizes a shared projector to translate both modalities into language-like embeddings that the LLM can comprehend.
  • Knowledge-Guided Instruction Data Generation: To train the model to "show its work," the authors built a pipeline that extracts 14 physiological feature sequences (such as Heart Rate, PR Interval, QRS Duration, and ST Form) across all 12 leads using a deterministic, rule-based tool (FeatureDB). A cardiology-specific Diagnosis Guider structures these measurements into prompts for GPT-4o, which then curates a highly detailed, grounded dataset called ECG-Grounding.

Datasets

The model was trained and evaluated using a mix of massive existing datasets and a newly proposed benchmark:

  • Training Data:
  • ECG-Instruct (from PULSE): Contains 1,156,110 standard ECG-text conversations.
  • ECG-Grounding (New): 30,000 fine-grained instruction-response pairs generated from the MIMIC-IV-ECG database, annotated with heartbeat-level physiological features.
  • Evaluation Benchmarks:
  • Grounded ECG Understanding Task (New): A clinically motivated benchmark designed to test a model's ability to identify diagnostic clues across multiple leads. Evaluated on test sets from MIMIC-IV-ECG (in-domain) and PTB-XL (out-domain).
  • ECG-Bench: An existing benchmark used to evaluate cardiac abnormality detection and report generation across diverse datasets (PTB-XL, CPSC2018, CODE-15%, CSN, and G12EC).

Key Insights and Ideas

  • The Power of "Time Series + Image": Time series models excel at capturing dynamic, heartbeat-to-heartbeat changes but can miss global spatial structures. Vision models see the big picture but overlook subtle temporal details. Fusing them creates a synergistic effect that mirrors a clinician's actual workflow.
  • Bootstrapping Grounding without Human Costs: High-granularity clinical data annotation is incredibly expensive. By using deterministic signal processing algorithms to extract exact wave measurements first, the authors successfully guided a frontier LLM (GPT-4o) to generate highly accurate, complex medical training data without manual expert labeling.
  • One-Step Training Efficiency: Unlike standard MLLM pipelines that use a multi-stage training process (aligning projectors first, then fine-tuning the LLM), GEM uses a one-step approach. Freezing the encoders while jointly training the projectors and the LLM enhanced cross-modal consistency, even with limited data.

Key Results

GEM demonstrated substantial improvements over state-of-the-art models like PULSE across all primary evaluation metrics.

Grounded ECG Understanding

On the newly introduced Grounded ECG Understanding task, GEM achieved massive leaps in explainability and lead coverage:

Table 1: Grounded ECG Understanding results on MIMIC-IV-ECG and PTB-XL.
  • Diagnosis Accuracy: Reached 87.24% on MIMIC-IV-ECG (outperforming PULSE's 81.14%) and 73.59% on the out-domain PTB-XL dataset (outperforming PULSE's 59.24%), proving strong generalization capabilities.
  • Lead Assessment Coverage: Jumped from a meager ~7-11% (PULSE) to over 71-79% (GEM), showing that GEM systematically evaluates almost all 12 leads rather than fixating on a few.
  • ECG Feature Grounding: Improved by roughly 25%, scoring around 75 out of 100, meaning the majority of GEM's diagnostic conclusions were explicitly linked to measurable ECG parameters.

Abnormality Detection

On standard classification tasks within ECG-Bench, GEM consistently achieved top-tier performance, notably hitting an accuracy of 92.6% on the CSN dataset.

Table 2: ECG-Bench abnormality detection results.

Limitations and Future Work

Despite its success, the authors note several limitations that must be addressed before clinical deployment:

  • Encoder Deficiencies: Subtle morphological patterns, such as minor ST-segment changes or P-wave abnormalities, were occasionally missed by the frozen encoders during the representation stage, leading to incorrect diagnoses.
  • Overstating Severity: GEM sometimes exaggerated the severity of certain conditions. Because it interprets ECGs in isolation without access to a patient's broader clinical history or context, it can lack the nuance a physician naturally applies.
  • Data Generation Alignment: While GPT-4o generated high-quality training data, it still occasionally produced target answers that misaligned with expert cardiologist interpretations.

Future Directions: The team plans to explore incorporating expert human feedback directly into the data generation loop (e.g., RLHF) and integrating patient clinical history to refine the model's safety and clinical alignment.

Summary updated

Abstract

While recent multimodal large language models (MLLMs) have advanced automated ECG interpretation, they still face two key limitations: (1) insufficient multimodal synergy between time series signals and visual ECG representations, and (2) limited explainability in linking diagnoses to granular waveform evidence. We introduce GEM, the first MLLM unifying ECG time series, 12-lead ECG images and text for grounded and clinician-aligned ECG interpretation. GEM enables feature-grounded analysis, evidence-driven reasoning, and a clinician-like diagnostic process through three core innovations: a dual-encoder framework extracting complementary time series and image features, cross-modal alignment for effective multimodal understanding, and knowledge-guided instruction generation for generating high-granularity grounding data (ECG-Grounding) linking diagnoses to measurable parameters ($e.g.$, QRS/PR Intervals). Additionally, we propose the Grounded ECG Understanding task, a clinically motivated benchmark designed to comprehensively assess the MLLM's capability in grounded ECG understanding. Experimental results on both existing and our proposed benchmarks show GEM significantly improves predictive performance (CSN $7.4\% \uparrow$), explainability ($22.7\% \uparrow$), and grounding ($24.8\% \uparrow$), making it more suitable for real-world clinical applications. GitHub repository: https://github.com/lanxiang1017/GEM.git
BibTeX
@inproceedings{lan2025gem,
  author = {Xiang Lan and Feng Wu and Kai He and Qinghao Zhao and Shenda Hong and Mengling Feng},
  title = {{GEM: Empowering MLLM for Grounded ECG Understanding with Time Series and Images}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2025},
  url = {https://openreview.net/forum?id=idtZwmjakN},
}
Figure