When a model explains an electrocardiogram (ECG), the recording of the heart's electrical activity, it makes physical claims about the waveform. It might write "PR interval prolonged," "T-wave inverted in V2–V4," or "baseline wander in lead III." The current way to check such claims automatically is to ask a large language model (LLM) to act as a judge. That judge writes and runs code: signal-processing tools measure the waveform, and the claim is compared against the numbers. This turns hand-crafted feature engineering, the decades-old practice of summarizing a signal with a few expert-defined measurements, into the judge's eyes. Feature engineering has a well-known limitation: it can only see what someone thought to measure. Our argument is that the judge inherits this limitation. A verifier built on measurement tools can only rule on claims expressible in its feature vocabulary. Explanations of ECGs are full of claims about shape, noise and localized morphology that fall outside that vocabulary. Such claims either go unchecked or get checked against the wrong proxy.
We propose to measure this ceiling directly and then break through it. We would build a per-claim verifier that has two parts. Claims a trusted feature exists for are routed to deterministic measurement. Claims outside that vocabulary go to a learned model that looks at the raw waveform. The learned model is trained on cardiologist-written statements and noise annotations rather than on computed features. The result we expect is an evaluator that rules on substantially more of the claims in real model explanations and stays accurate on them. We also expect its disagreements with feature-only judges to fall in the categories where hand-crafted features are known to be weak.
Core Claim
The closest existing work is ECG ReasonEval, introduced in How Well Do Multimodal Models Reason on ECG Signals?. It splits ECG reasoning into two parts. Its perception pipeline has a Claude-powered agent run wave-delineation models in Python against the waveform. Its deduction pipeline uses retrieval to check clinical logic. It reports that perception scores correlate only weakly with diagnostic accuracy (r = 0.50, against r = 0.88 for deduction). That paper motivates our proposal: it shows models invent waveform evidence (an "illusion of thinking") and that perception is the poorly understood axis. It does not ask what fraction of claims its perception pipeline can actually verify, or whether it errs systematically on certain kinds of claims.
Our claim is that verification coverage and accuracy depend on claim type. They should follow the same categories along which Time-Series Foundation Model Embeddings as Means for Physiological Feature Extraction found learned representations helping or failing. That work found:
- periodicity features improved 26.02% relative RMSE;
- amplitude features improved 18.28%, with much of that recovered only by re-injecting global scale statistics;
- second-derivative morphology improved just 2.60%.
We hypothesize that feature-based judges are strong on periodicity and interval claims and blind on morphology and noise claims. We expect a naively learned judge to be blind on amplitude claims, because instance normalization (rescaling each signal to zero mean and unit variance) strips absolute voltage. A judge that routes each claim by type should beat either alone. The new contributions are the measured coverage ceiling and the routed verifier. The judging idea itself is not new.
Research Program
The first component is a claim corpus. We would generate explanation traces for PTB-XL records (PTB-XL) from three sources:
- ECG-Chat, which aligns a contrastive waveform encoder to Vicuna-13B;
- ECG-LM, which aligns a ResNet encoder to a 7B biomedical LLM;
- a generalist LLM given the signal as text.
These systems were evaluated mainly with BLEU/ROUGE and label matching, so their waveform justifications have never been checked claim by claim. An LLM would split each trace into atomic waveform claims. We would sort the claims into five categories: interval/periodicity, amplitude, rhythm, morphology (shape, notching, localized ST/T changes) and signal quality (noise, drift, artifacts). Cardiologists would label a held-out set of roughly 1,500 claims as supported, contradicted, or not determinable from the recording. This set is the ground truth. No reported result found on Lacuna gives claim-level accuracy for any judge on real ECG explanations, so this set fills a real gap.
The second component is the feature route. PTB-XL+ supplies the same measurements from three independent extractors: Marquette 12SL, Uni-G and ECGDeli. A claim mapped to a feature is checked against it. A verdict is issued only when the three extractors agree within a tolerance. PTB-XL+ reports high agreement on global features (correlation above 0.9) but notable disagreement on subtle components such as secondary R′/S′ waves. Cross-extractor agreement is therefore a free, principled abstention signal. That signal is the specific use of hand-crafted features this proposal makes.
The third component is the learned route: a waveform-and-claim verifier.
- Positive claims come from PTB-XL's cardiologist form, rhythm and noise annotations, converted into templated and paraphrased sentences.
- Hard negatives are minimal perturbations of those claims: the wrong lead, the opposite polarity, a shifted location, or a swapped noise type.
- Extra pairs come from ECG-QA (ECG-QA) Verify questions turned into declarative claims.
The verifier itself borrows a method from Rethinking LLM-as-a-Judge. That paper's INSPECTOR probes trained on a small model's hidden states judged text quality better than prompting the same model (over 20% F1 gain, 80–90% binary accuracy). We would train such probes on the hidden states of the ECG-language model while it reads the waveform plus the claim. As an alternative, we would train a small cross-encoder on a waveform encoder. To address the amplitude blind spot, we would concatenate per-lead mean and standard deviation, following Lee et al. A router sends each claim to the feature route if a trusted feature exists and the extractors agree, and to the learned route otherwise.
Baselines
The naive reference point is surface overlap: ROUGE between the explanation and the PTB-XL report, plus exact match of the final label. This is what ECG-LM and ECG-Chat relied on. We expect it to detect essentially none of the invented evidence, and including it anchors how much any claim-level judge adds.
The strongest existing method is the ECG ReasonEval perception pipeline. It is an LLM agent that writes code calling delineation models and compares outputs to claims. It is the only published system that checks waveform claims against the signal. We would run it with its published prompts on exactly our claim set. If the proprietary agent model is unavailable, we would use the strongest available frontier model and report the substitution. Its only reported numbers are correlations with diagnostic accuracy (perception r = 0.50, deduction r = 0.88). No reported result found on Lacuna gives its claim-level accuracy or coverage, so we would measure both. Beating it means higher coverage at equal or better precision on cardiologist labels. In particular, it means significantly higher macro-F1 on morphology and signal-quality claims, with no loss on interval claims, at a fraction of its inference cost.
The deterministic feature-only verifier is the PTB-XL+ route alone, with extractor-agreement abstention. It is the strongest cheap competitor. On interval and rhythm claims we expect it to match or beat everything. PTB-XL+ already shows a Random Forest on Uni-G features reaching 0.875 macro AUC, and an ECGDeli Random Forest beating CNNs on rhythm. Beating it means more total claims verified correctly, not better interval accuracy.
The statement-classifier baseline maps each claim to its nearest PTB-XL statement and reads off a standard 71-statement waveform classifier's prediction. It shares the learned route's training data and compute budget, and its hyperparameter search would be tuned to the same budget. It tests whether claim conditioning adds anything over a label classifier.
The prompted small-model judge prompts the same ECG-language model used for probing to output a verdict. This follows the INSPECTOR comparison.
Two ablations isolate our ingredients. The first removes routing and sends every claim to the learned verifier, with and without the re-injected scale statistics. This tests the amplitude hypothesis directly. The second removes extractor-agreement abstention, which tests whether hand-crafted features are useful mainly as a reliability signal.
Why not just…?
Why not just check every claim against the commercial feature extractors? For claims about intervals, rates and axis, this is the right answer, and our router uses it. The open question is how many claims remain. Explanations routinely describe localized morphology and artifacts with no corresponding scalar in 12SL, Uni-G or ECGDeli. PTB-XL+ itself documents extractor disagreement on subtle waveform components. The coverage measurement in our first experiment settles this. If over 80% of real claims are feature-checkable with high agreement, the learned route is unnecessary. In that case the paper's contribution reduces to the coverage audit, and we would say so.
Why not just run the ReasonEval agent and let it write more code? An agent can in principle write a morphology detector on the fly. But whatever code it writes is still a hand-crafted feature, invented per claim and unvalidated. It also costs frontier-model inference per claim. The decisive comparison is per-category accuracy against cardiologist labels. If the agent matches our verifier on morphology and noise claims, our verifier's remaining advantage is cost alone, and we would report it that way.
Why not just classify the ECG and match claims to labels? This is the most serious threat, because our learned route trains on the same statements. It fails by construction on claims that are lead-specific, localized or compositional, such as "inverted T in V2–V4 but upright in V5." It also cannot say "not determinable." ECG-QA shows that compositional comparison questions drop to roughly 20–22% exact match even for the best model. If the statement-classifier baseline ties our verifier on the cardiologist set, we would adopt it as the learned route and drop claim conditioning.
Why not just convert the waveform to text features and ask a strong LLM? ECG-QA tested this two-step pipeline. LLMs fed intermediate text descriptions did not surpass end-to-end multimodal models such as M3AE (74.6% exact match on Single-Verify). TSQueryBench (TSQueryBench) finds LLM judges over numeric series miss variance and noise changes even when they rank first-order changes well (0.84 ranking accuracy for Qwen-3 8B). Both results predict failure on exactly the signal-quality claims we target. We would include this pipeline as a further comparison on the signal-quality category to confirm it.
Proposed Experiments
Coverage audit. The first experiment reports what fraction of atomic claims each verifier will rule on, broken down by the five categories and by source model. The decisive pattern we predict has two parts. Feature-only and agent verifiers should leave a large share of morphology and signal-quality claims uncovered or wrong. The learned route without scale statistics should fail on amplitude claims, mirroring Lee et al. If the categories do not separate this way, the core hypothesis is wrong.
Accuracy against cardiologists. The second experiment reports macro-F1 and coverage–precision curves on the cardiologist set for all baselines and ablations under equal training data and tuning budgets. Confidence intervals would come from bootstrapping over records rather than claims, since claims from one ECG are correlated.
Bias stress test. The third experiment borrows perturbations from CALM (Justice or Prejudice?). Each false claim is rewritten with an authoritative guideline citation, padded with extra words, or phrased politely. We then measure how often verdicts flip. We expect the feature and learned routes, which never read the surrounding rhetoric, to be far more robust than prompted judges. That would be a concrete reason to verify per claim against the signal.
Model ranking. The fourth experiment scores ECG-Chat, ECG-LM and a generalist model with each judge. It asks whether the routed verifier changes the perception ranking ReasonEval reports, and whether its perception score correlates with diagnostic accuracy more strongly than the r = 0.50 reported there.
Transfer. A final transfer check applies the PTB-XL-trained verifier to MIMIC-IV-ECG records with their machine reports. The brittleness reported for clinical reward models in Brittleness and Promise warns that learned judges may not move across sites. If accuracy collapses there while the feature route holds, that finding constrains where learned waveform judges can be trusted.
References
- How Well Do Multimodal Models Reason on ECG Signals?
- Time-Series Foundation Model Embeddings as Means for Physiological Feature Extraction
- PTB-XL, a large publicly available electrocardiography dataset
- ECG-Chat: A Large ECG-Language Model for Cardiac Disease Diagnosis
- ECG-LM: Understanding Electrocardiogram with a Large Language Model
- PTB-XL+, a comprehensive electrocardiographic feature dataset
- ECG-QA: A Comprehensive Question Answering Dataset Combined With Electrocardiogram
- Rethinking LLM-as-a-Judge: Representation-as-a-Judge with Small Language Models via Semantic Capacity Asymmetry
- TSQueryBench: LLM-as-a-Judge for Time-Series Explanations
- Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge
- Brittleness and Promise: Knowledge Graph–Based Reward Modeling for Diagnostic Reasoning