Automatic metrics for generated radiology reports now mostly work by pulling clinical findings out of the generated report and the reference report, then counting how many findings agree. The best-known example, RadGraph F1, extracts each report into a small graph of anatomy and observations and scores how much the two graphs overlap. The matching step underneath is close to literal string identity. "Cardiac enlargement" does not match "cardiomegaly." A generated report that says "consolidation" where the radiologist wrote "lobar pneumonia" is scored exactly like one that invents a finding that was never there. Machine translation hit the same wall twenty years ago under the name out-of-vocabulary bottleneck, meaning the system has no entry for a term it meets at test time. That literature found that strict synonymy covers too little. Allowing directional substitution, where a specific term may be replaced by a more general one it entails, recovered far more cases at usable precision.
We propose to bring that directional idea into clinical graph metrics. When a generated finding fails to match the reference exactly, the metric decides which way the two terms relate. If they are equivalent, the finding gets full credit. If the generated term is a correct but vaguer generalization of the reference, it gets partial credit. If the generated term is more specific than the evidence supports, or names a sibling or opposite finding, it is treated as an error. The term-pair relations are computed once over the finite vocabulary of extracted entities and stored. The metric therefore keeps the speed and auditability of RadGraph F1 while addressing its main lexical blind spot. A second contribution may matter as much as the metric: a controlled substitution benchmark that measures, for any report metric, how it treats synonyms, generalizations, over-specifications, sibling swaps and polarity flips.
Core Claim
The lexical failures of graph-based clinical metrics are not symmetric, and fixing them with a symmetric similarity score is the wrong repair. Clinically, under-specification ("opacity" for "pneumonia") is a lossy but safe omission of detail. Over-specification ("pneumonia" for "opacity") is an unsupported diagnostic claim, much closer to a hallucination. Embedding similarity gives both directions the same score. Exact matching gives both a zero. We claim that a metric which keeps RadGraph's structured extraction but replaces exact entity matching with direction-aware graded matching has two properties. First, on targeted substitution tests it ranks under- and over-specification in the order radiologists do, where symmetric soft matching and exact matching cannot. Second, it closes a meaningful part of the gap between RadGraph F1 and distilled LLM judges in correlation with radiologist error counts on ReXVal, at a small fraction of their per-report cost. We do not claim that it will beat the best LLM judge on aggregate correlation; that is an empirical question the experiments settle.
Where This Comes From
Three graph or tabular metric families motivate the work. RadGraph defined the schema of anatomy and observation entities, where each observation is labeled present, uncertain or absent and linked by located-at, suggestive-of and modify relations. Its extractor reaches 0.94 entity F1 in-distribution. Evaluating Progress in Automatic Chest X-Ray Radiology Report Generation turned that schema into RadGraph F1 and the composite RadCliQ. RadGraph-XL extended the schema to CT and MRI reports, where the vocabulary grows considerably. All three match extracted entities by span identity. That matching rule is the component we replace. We keep their extractors unchanged, and the polarity labels in particular, because they already separate "no pneumothorax" from "pneumothorax" outside the lexical match. CLEAR avoids free-form matching by mapping reports onto 13 fixed conditions. That buys interpretability but leaves findings outside those conditions unscored, and it falls back to surface similarity or a proprietary model for location and recommendation text. Our approach is open-vocabulary by design.
The method comes from the out-of-vocabulary translation literature. Source-Language Entailment Modeling for Translating Unknown Terms replaced unknown terms with entailed, more general terms from a taxonomy. Coverage of sentences with unknown terms rose from about 32% to over 48%, at 71–73% substitution precision. The paper also notes the cost: generalization throws away specificity. We take both points. Direction is the useful signal, and the specificity loss should be priced as partial credit rather than ignored. Graph Propagation for Paraphrasing Out-of-Vocabulary Words induced mappings for unseen terms by passing labels through a graph built from monolingual text, including on the medical EMEA corpus, where BLEU went from 20.34 to 20.80. We reuse that idea as one way to propose candidate relations for rare entities.
Two negative results constrain the design. Improved SMT Using Monolingually-Derived Paraphrases documents that context-based similarity confuses antonyms, because opposites occur in the same contexts. The Razmara et al. propagation paper reports that meaning drifts over multi-step propagation paths. Together these rule out using raw distributional similarity as the final arbiter of a match, so propagation only proposes candidates. Distilled judges such as GREEN and LLM-RadJudge already handle common synonyms. They are the bar to clear, and their cost and opacity are the reason a structured alternative is worth having.
Research Program
The metric has three parts. Extraction is unchanged: the RadGraph-XL extractor runs on candidate and reference reports. Every extracted observation and anatomy span is normalized by lowercasing and lemmatizing, and is added to a global entity vocabulary. Over MIMIC-CXR the RadGraph inference set already contains roughly six million entities, so this vocabulary is large but finite and heavy-tailed.
The relation lexicon is built offline. For each pair of entity strings that co-occur as unmatched candidate–reference pairs in practice, meaning they attach to the same anatomy in aligned graphs, we assign one of five relations. The candidate can be equivalent to the reference, a generalization of it, a specialization of it, a sibling (a different finding at the same level), or incompatible with it. Candidate pairs come from three sources, which are compared as ablations. The first is distributional neighbors over the MIMIC-CXR report corpus, propagated through a tripartite graph in the style of Razmara et al. The second is neighbors in the RadGraph relation structure itself; for example, suggestive-of edges such as "opacity → pneumonia" provide directional evidence. The third is an instruction-tuned language model queried once per term pair. A radiologist audits a stratified sample of the lexicon, and the audited precision of each source is reported. Because the lexicon is computed per term pair rather than per report pair, the expensive judgment is paid once and reused across every evaluation.
Scoring then works on graph alignment. Write $c$ for a candidate entity and $r$ for a reference entity attached to the same anatomy with the same polarity label. Their match value is
$$m(c,r) = \begin{cases} 1 & \text{equivalent} \\ \alpha & c \text{ generalizes } r \\ -\beta & c \text{ specializes } r,\ \text{sibling, or incompatible} \\ 0 & \text{no relation found} \end{cases}$$
Here $\alpha \in (0,1)$ is the credit for a correct but vaguer term and $\beta \ge 0$ is the extra penalty for an unsupported or wrong term. We align entities greedily to maximize total match. Precision and recall over entities and relations are then computed with these graded values in place of 0/1 counts. We fit only the two scalars $\alpha$ and $\beta$, by maximizing Kendall's tau on a held-out development split. This keeps the metric interpretable: every point lost traces back to a named entity pair and a named relation.
Baselines
The naive reference point is exact-match RadGraph F1 as defined by Yu et al. We run it with the same extractor checkpoint we use, so any difference comes only from the matching rule. LLM-RadJudge reports RadGraph F1 Kendall's tau of 0.19–0.60 against radiologists on ReXVal. Any gain must be measured against our own re-run of this baseline on identical splits.
The ablation that isolates the key ingredient is symmetric soft matching. Here the same extractor and alignment are used, but $m(c,r)$ is the cosine similarity of clinical-BERT embeddings of the two spans, thresholded and with its threshold tuned on the same development split. This is the most direct competitor to our claim. It shares everything except directionality. If it matches our metric on both ReXVal and the substitution benchmark, directionality is not doing the work. A second ablation sets $\beta=0$, keeping direction-aware credit but removing the over-specification penalty. This tests whether the asymmetry, rather than just broader coverage, drives agreement with radiologists.
The strongest existing method is the distilled LLM-RadJudge evaluator (BioMistral-7B-ft). It reports Kendall's tau of 0.7487 with the six-radiologist consensus on ReXVal, above GPT-4's 0.7348. GREEN reports 0.63–0.64, at about 1.06 seconds per sample on an A100. We also include RadCliQ, which reports Kendall tau-b of 0.522 in Yu et al. We run released checkpoints where available, on the same ReXVal pairs and substitution items, and report wall-clock cost per report on the same GPU. Beating the strongest method would mean exceeding 0.7487 on ReXVal. We consider that unlikely and would not claim it without a significant difference under paired bootstrap. The success criterion we commit to is twofold. First, a statistically significant improvement over exact-match RadGraph F1 and over symmetric soft matching on ReXVal. Second, higher accuracy than GREEN and LLM-RadJudge on the over-specification and sibling-swap items of the substitution benchmark, at RadGraph-level cost.
Why not just…?
Why not just use embedding similarity to soft-match entities? This is the real threat, and the proposal is built around testing it. Within RadGraph the polarity label is stored separately from the entity string, so the antonym problem documented by Marton et al. is partly neutralized for presence and absence. Embedding matching may therefore handle plain synonyms well. What it cannot do is tell direction, because cosine similarity is symmetric. Yu et al. also found that BERTScore, applied to whole reports, struggled to rank real generations above a random baseline. The symmetric-matching ablation together with the generalization-versus-specialization split of the substitution benchmark decides the question. If the ablation does as well on those items, we will report that directionality adds nothing.
Why not just use GREEN or LLM-RadJudge, which already understand synonyms? On aggregate ReXVal correlation they are likely to stay ahead, and we say so. Their weaknesses are specific. They take about a second per report, which rules them out as an inner-loop training signal. They give no per-entity audit trail, which CLEAR also criticizes in single-score judges. And they are frozen after distillation, so rare or institution-specific terms may slip through. The substitution benchmark, restricted to low-frequency entities, measures that last point directly. If the distilled judges already penalize over-specification correctly there, the case for our metric narrows to cost and interpretability, and we would present it on those grounds alone.
Why not just map everything to a fixed concept list like CheXpert's 14 labels? CLEAR reports that its own label extractor beats CheXbert and CheXpert by 15.8% average accuracy on positive conditions. That indicates fixed labelers lose information even within their own label set. By construction they cannot score findings outside it, nor location or severity. A fixed medical ontology would be a reasonable source of generalization relations, and it fits into our lexicon-building step as one more candidate source. It does not replace graded, direction-aware matching.
Proposed Experiments
The first experiment is the substitution benchmark. From MIMIC-CXR test reports we sample findings and create controlled variants by editing one entity span. A variant uses a synonym, a correct generalization, an over-specification, a sibling finding, or a polarity flip. A frequency stratum separates common entities from those rare in the RadGraph training data. Board-certified radiologists rate a subset of variants for clinical severity of the change, giving a reference ordering. Each metric is scored on how often it orders variant pairs the way radiologists do. The decisive check is the generalization-versus-over-specification contrast: our metric should separate them, while exact matching and symmetric soft matching should not.
The second experiment is correlation with radiologists on ReXVal. Kendall's tau with the six-radiologist error counts is computed for every baseline and ablation, with paired bootstrap confidence intervals. $\alpha$ and $\beta$ are fit on a held-out fold so the reported numbers are not tuned on the test pairs.
The third experiment tests the multi-modality vocabulary on the RadGraph-XL CT and MRI reports, where the lexicon is broader. We measure how often unmatched candidate–reference entity pairs receive a non-zero relation, and the audited precision of those relations. We also measure how often the extractor misses an out-of-vocabulary finding entirely. Matching cannot repair extraction failures, so that rate bounds what the method can deliver and is reported separately.
Finally, we report per-report scoring time against GREEN's 1.06 seconds, to check that amortizing the lexicon makes the metric usable as a training-time reward.
References
- RadGraph: Extracting Clinical Entities and Relations from Radiology Reports
- Evaluating progress in automatic chest X-ray radiology report generation
- RadGraph-XL: A Large-Scale Expert-Annotated Dataset for Entity and Relation Extraction from Radiology Reports
- CLEAR: A Clinically Grounded Tabular Framework for Radiology Report Evaluation
- Source-Language Entailment Modeling for Translating Unknown Terms
- Graph Propagation for Paraphrasing Out-of-Vocabulary Words in Statistical Machine Translation
- Improved Statistical Machine Translation Using Monolingually-Derived Paraphrases
- GREEN: Generative Radiology Report Evaluation and Error Notation
- LLM-RadJudge: Achieving Radiologist-Level Evaluation for X-Ray Report Generation