RAG-TRACE: Tracing Measurement Evidence and Diagnostic Completeness in AI-Generated ECG Reports

Elyas Irankhah, Shahed Salehzehi, Parisa HosseiniarXiv 2026 · arXiv

TL;DR

RAG-TRACE is a new framework designed to ensure that AI-generated medical reports are actually backed by patient-specific data. By applying deterministic verification rules to Retrieval-Augmented Generation (RAG), the researchers found they could strip away the "hallucinated" clinical criteria that LLMs often include, significantly improving report reliability without compromising diagnostic accuracy.

What Problem Does This Paper Address?

Large Language Models (LLMs) are increasingly being tested for generating clinical narratives, such as electrocardiogram (ECG) reports. To improve accuracy, researchers often use Retrieval-Augmented Generation (RAG), which allows the model to look up external medical knowledge—like the specific criteria for a heart attack—before writing.

However, there is a "verifiability gap." A RAG system might retrieve a perfectly valid clinical criterion (e.g., "QRS duration ≥ 120 ms") because it is relevant to a suspected diagnosis, but the LLM might then claim this criterion is met even if the patient's actual measurements don't support it. In a medical context, these "unsupported applications" are dangerous. The authors set out to measure how often RAG introduces these errors and whether they can be automatically corrected.

Methods

The authors developed RAG-TRACE, a pipeline that combines LLM generation with a deterministic verification layer.

  1. Criteria Library: They built a library of 47 "criteria cards." Some were testable (containing numeric thresholds like heart rate or voltage), while others were untestable (descriptive notes like "lead reversal" or "age").
  2. Report Generation: They used models like GPT-4.1-mini and Qwen3-8B to generate structured ECG reports. They compared three settings: no retrieval, keyword retrieval, and TF-IDF retrieval.
  3. Deterministic Verification: This is the core of RAG-TRACE. The system extracts the criteria the LLM claimed to use and checks them against raw patient measurements (using the Marquette 12SL algorithm). If a claim isn't supported by the numbers, it is withdrawn.
  4. Independent Assessment: To ensure the verification wasn't biased, they used a second, independent measurement algorithm (Uni-G) to cross-check the results.
The RAG-TRACE pipeline: from data preparation to measurement-based verification.

Datasets

  • PTB-XL: The primary dataset, containing 21,799 clinical ECG recordings from 18,869 patients.
  • PTB-XL+: Used to access the specific feature tables and measurements (intervals, axes, voltages) required for verification.
  • MIMIC-IV-ECG: A secondary dataset used for waveform network specificity controls to ensure the verification rules generalized well.

Key Insights and Ideas

  • Relevance is not Evidence: RAG is excellent at finding relevant information, but it lacks the logic to ensure that information applies to the specific patient at hand. This leads to a high rate of "untestable" or "disconfirmed" criteria in AI reports.
  • The "Release Score" Danger: The authors introduced a "release score" to track unsupported applications. They found that including descriptive, non-numeric cards in a RAG library significantly inflates this score, as LLMs struggle to ground qualitative descriptions in quantitative data.
  • Verification is "Safe": A major finding was that you can automatically withdraw unsupported criteria without hurting the overall diagnostic conclusion. The model's ability to identify conditions like hypertrophy (recall) remained stable even after the "fluff" was removed.
  • Model Variability: Different models fail in different ways. For example, Qwen3-8B showed a distinct failure pattern where it incorrectly applied measurable criteria that were then rejected by the independent Uni-G algorithm.

Key Results

  • High Error Rates in RAG: Keyword retrieval introduced unsupported or untestable criteria in 46.8% of GPT-4.1-mini reports.
  • Effective Filtering: The verification rule successfully withdrew 647 criterion applications. Crucially, the independent Uni-G algorithm agreed with these withdrawals in nearly every case (preserving 268 out of 269 Uni-G supported applications).
  • Stable Diagnostics: In a controlled experiment, while retrieval increased the rate of unsupported criteria from 17.4% to 31.1%, the actual recall for hypertrophy remained identical at 62.6%.
  • Library Impact: Restricting the RAG system to only the 14 testable (numeric) criteria slashed the unsupported-criterion rate from 46.8% down to 13.8%.
Comparison of keyword retrieval vs. post-verification results across different models.

Limitations and Future Work

The authors acknowledge several limitations:

  • Ground Truth: The study relied on existing PTB-XL labels rather than a new, independent clinical review of the AI-generated narratives.
  • Sentence Extraction: Their "label-aware" analysis for diagnostic agreement had to exclude some sentences where the automated judges couldn't agree on the meaning.
  • Scope: The library was limited to 47 cards; real-world cardiology involves hundreds of nuanced criteria.

Future Work: The team suggests moving toward "measurement-aware retrieval," where the RAG system only retrieves criteria that the patient's data actually satisfies, rather than retrieving based on keyword relevance alone. They also aim to extend RAG-TRACE to other physiological signals like EEG or sleep studies.

Summary updated

Abstract

Purpose: Retrieval-augmented generation (RAG) can introduce electrocardiogram (ECG) criteria that lack patient-specific measurement support. This study evaluates the reliability of retrieved diagnostic criteria in AI-generated ECG reports. Methods: We analyzed 1,083 human-validated PTB-XL patient records using a library of 47 criteria cards, including 33 without numeric verification rules. Language models generated structured reports with and without retrieval. Criterion applicability was evaluated using Marquette 12SL measurements, with independent assessment using Uni-G and comparison against expert-validated diagnostic labels. Results: Keyword retrieval introduced untestable or measurement-disconfirmed criteria in 46.8% of GPT-4.1-mini reports. A deterministic verification rule withdrew 647 criterion applications, including 639 without numeric checks, while preserving 268 of 269 applications independently supported by Uni-G. Restricting retrieval to 14 testable criteria reduced the initial unsupported-criterion rate to 13.8%. Under matched instructions and an identical output schema, retrieval increased unsupported-criterion reports from 17.4% to 31.1%, while hypertrophy recall remained unchanged at 62.6%. Conclusion: These findings identify a fundamental limitation of retrieval-based clinical report generation: retrieved criteria do not necessarily provide valid patient-specific evidence. Measurement-based verification can reduce unsupported criterion applications, but diagnostic validity requires independent assessment.
BibTeX
@misc{irankhah2026rag,
  author = {Elyas Irankhah and Shahed Salehzehi and Parisa Hosseini},
  title = {{RAG-TRACE: Tracing Measurement Evidence and Diagnostic Completeness in AI-Generated ECG Reports}},
  year = {2026},
  eprint = {2611.01409},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.01409},
}
Figure