Automatic radiology report generators are now built to write in Spanish, Chinese and other languages, but the tools that check whether a report is medically correct work only on English text. Multilingual systems therefore fall back on word-overlap scores such as BLEU, which the field already knows reward fluent hallucinations. This proposal builds a scorer that uses the X-ray image itself as the shared reference across languages. The scorer reads each finding in a generated report, records what was claimed, whether it was claimed present or absent, and which region of the image the model pointed to. It then matches those findings against the radiologist's findings by concept and by anatomical location.
Because the image does not change when the language or the hospital's writing habits change, this score should stay the same when a correct finding is rephrased in another language or in clipped bedside shorthand. It should drop when a finding is negated, invented, missed, or placed in the wrong part of the chest. We will test both properties on PadChest-GR, a Spanish–English chest X-ray dataset in which every positive finding sentence is tied to a radiologist-drawn box. We will also collect a small Spanish/English radiologist error-count study so the scorer is validated against experts rather than against another metric.
Core Claim
Grounded report generation produces reports in which each finding sentence comes with a bounding box. For these reports, a clinical scorer should do its matching at the level of findings, and matching on concept, assertion and anatomical region together should beat two simpler options. The first is translating the report to English and applying an English clinical metric. The second is adding that text score to a separate box-overlap score. We make two predictions. Joint matching will correlate with radiologist error counts about equally for Spanish and English candidates, while translate-then-score methods lose correlation on Spanish. Joint matching will also catch a class of errors that decoupled scores cannot separate: right concept, wrong place. Its sharpest form is a report with several findings of the same type, such as bilateral effusions or multiple nodules, where a separate box-overlap average cannot tell which box belongs to which sentence.
Where This Sits
The motivating evidence is well established. Lexical metrics do not track correctness of medical observations (Jing et al.; Lanz et al.). On the ReXVal expert set, BLEU reaches a Kendall's tau of only 0.35 with radiologist error counts. The English clinical metrics replace word overlap with structure. RadGraph parses reports into anatomy and observation graphs with present/uncertain/absent labels. RadCliQ regresses BLEU and RadGraph F1 onto expert error counts. GREEN distills GPT-4 error counting into small local models. ICARE has two LLM agents quiz each other about the reports. All four compare text to text, in English only, and never look at the image.
On the multilingual side, M³FM generates English and Chinese reports but reports BLEU-4 and ROUGE-L. That is exactly the gap the clinical-metric literature warns about.
The closest prior work is van Sonsbeek et al.. They map English and Spanish reports to language-independent medical concept codes from UMLS, the large biomedical vocabulary that assigns one identifier per concept, and use the resulting graphs for disease classification. They reach an average AUC of 0.967 on PadChest. We reuse their idea that concept codes make Spanish and English comparable, and use it as one of our concept normalizers. They build no evaluation metric, handle no generated text, and do no region-level grounding. Their finding that the graph model does not consistently beat ClinicalBERT on large English data also warns that hard ontology mapping alone is brittle.
PadChest-GR supplies the data: 4,555 studies, 7,037 box-linked positive sentences and 3,422 negative sentences, with native Spanish text and radiologist-verified English translations. It defines the grounded generation task but proposes no metric for it.
Two further lines of work constrain the design. Medical vision-language models often attend to the wrong image region (How Do Medical MLLMs Fail?). This is why location must be scored rather than assumed. End-to-end models also learn positional shortcuts instead of anatomy, while an explicit module predicting 36 thoracic landmarks reaches 0.73 IoU on anatomy grounding (AnatomiX). This is why we compare regions through anatomical landmarks rather than raw box overlap.
Finally, the clinical-sublanguage literature explains why text-side robustness matters even before switching languages. Clinical notes drop function words (Jiang et al.), are dense with ambiguous abbreviations (Grossman Liu et al.), and vary by institution (Taylor et al.). RadGraph extraction itself degrades moving from MIMIC-CXR to CheXpert.
Research Program
Each report is turned into a set of findings. A finding is a triple $(c, a, R)$: $c$ is a normalized clinical concept, $a$ is its assertion (present, uncertain or absent), and $R$ is the set of anatomical regions its box overlaps.
The regions come from a landmark detector in the style of AnatomiX, run once on the image. The detector is shared by the candidate and the reference, so region comparison is about anatomy and not box size. This matters because radiologists draw boxes inconsistently, and RadGraph's relation agreement between annotators was only a Cohen's kappa of 0.397.
The concept and assertion are produced by a multilingual clinical tagger trained on PadChest-GR's Spanish and English sentences against PadChest's finding vocabulary. We will compare it with a UMLS-mapping variant following van Sonsbeek et al.
Candidate and reference findings are paired by minimum-cost bipartite assignment. A pair counts as correct when
$$ c_i = c_j,\quad a_i = a_j,\quad R_i \cap R_j \neq \emptyset , $$
where $i$ indexes a candidate finding and $j$ a reference finding. The reported score is the F1 over correctly matched findings, so precision penalizes invented findings and recall penalizes missed ones.
Unmatched or partially matched pairs are labeled with an error type. A wrong assertion is a matching concept and region with flipped presence. A wrong location is a matching concept and assertion with no shared region. The remaining types are false findings and missed findings. These typed errors are the interpretable output, in the spirit of GREEN's error notation.
Negative findings in PadChest-GR have no box. They are matched on concept and assertion only. Reports without boxes can be scored in a text-only mode, which is the same matching with the region condition dropped. That mode is also the ablation that isolates the image's contribution.
To validate the scorer we need expert judgments, and none exist for Spanish. We will generate candidate grounded reports on the PadChest-GR test split from several systems: a fine-tuned grounded generator, a retrieval baseline, and a zero-shot multimodal model. We add controlled perturbations: negation flips, left–right swaps, box moves to an adjacent region, and deleted or invented findings.
Spanish-speaking radiologists will count errors on roughly 300 candidate–reference pairs, using ReXVal's error categories plus an explicit wrong-location category. Each pair is shown in both its native-Spanish and its English form so the two conditions can be compared directly. This study is the main cost of the project, and it is kept deliberately small, on the scale of ReXVal's 200 pairs.
Baselines
The do-nothing reference point is BLEU and ROUGE-L computed in the report's own language, which is how M³FM was evaluated. Everything should beat it. On English ReXVal, BLEU's Kendall's tau is 0.35 (GREEN).
The strongest text-only competitor is translate-then-GREEN. We translate the Spanish candidate with a strong general translator and score it with GREEN's released local model against the verified English reference. We run translate-then-RadGraph F1 the same way. These inherit GREEN's English validation: Kendall's tau 0.63–0.64 on ReXVal and 62–68% agreement with radiologist pairwise preferences, versus RadCliQ's tau of 0.522 (Evaluating Progress). There is no reported result on Lacuna for any metric's expert correlation on PadChest-GR or on Spanish reports, so all numbers there will be ours. To keep the comparison fair, every metric is scored on identical pairs and correlated against the same error counts. We run GREEN unmodified, and also with the radiologist-verified English translation substituted for machine translation. The second variant is an upper bound for translation quality and separates translation errors from metric errors.
The strongest competitor that uses location is a decoupled composite: translate-then-GREEN plus the mean box IoU of the candidate's boxes against the reference boxes. The two are combined by a linear regression fitted to expert error counts on a held-out split, exactly as RadCliQ combines its parts. Our scorer gets the same held-out split to set any weights. Beating this composite is the decisive test of joint matching.
We also include a zero-shot multimodal LLM judge given the image, both reports and the boxes. Its English text-only analogue matched radiologists only 23% of the time in GREEN's study.
Three ablations isolate the key ingredient. Text-only mode drops the region condition. Raw-IoU mode replaces the anatomical-region test with a box-overlap threshold. Single-language-tagger mode trains the concept tagger on English only.
Beating a baseline means a higher Kendall's tau with radiologist error counts, with bootstrap confidence intervals over pairs that do not overlap. The claim is about robustness, so we must win on the Spanish-candidate condition. A smaller gap between Spanish and English correlation also counts in our favor.
Why not just…?
Why not just translate to English and run GREEN? This is the most serious threat. PadChest's Spanish is radiology-department prose, not bedside shorthand, so modern translation may handle it well. If translate-then-GREEN matches our scorer on text errors in the Spanish condition, we will report that plainly and narrow the claim to location errors. The verified-translation variant tells us how much of any gap is due to translation. The shorthand-perturbation test (below) tells us whether the gap widens as text becomes more telegraphic, which the clinical-sublanguage literature predicts. What translation cannot do under any outcome is score boxes. The Flickr30k Entities JP work showed that translation breaks the link between phrases and regions even in everyday images.
Why not just add a box-IoU term to an existing text metric? This alternative is cheaper and may be good enough on single-finding reports. Its weakness is structural. Averaging IoU separately from text matching cannot check that the box attached to "right effusion" lies over the right lung rather than the left. With several findings of the same concept, it cannot tell which box goes with which sentence. We will report results separately on the subset of pairs with repeated concepts or bilateral findings, which is where joint assignment should matter. If the composite ties us there too, joint matching is not worth its complexity, and we would recommend the composite.
Why not just ask a multimodal LLM to judge? GREEN found that zero-shot GPT-4 matched radiologists only 23% of the time on English text comparisons. Medical multimodal models also attend to the wrong regions (How Do Medical MLLMs Fail?), so giving the judge the image may not help it check location. The judge also cannot run inside a hospital firewall. We include it as a baseline rather than assume it fails.
Proposed Experiments
The first experiment is expert correlation on PadChest-GR. We report Kendall's tau between each metric and radiologist error counts for Spanish candidates, English candidates, and the location-error subset. As an English sanity check, we run text-only mode on ReXVal, using the reports' text alone, against GREEN's reported 0.63–0.64. The point is to confirm that our concept tagger does not lose ground on English before we claim anything multilingual.
The second experiment is an invariance-and-sensitivity suite built from PadChest-GR's parallel sentences. For invariance, we take a correct finding and render it in native Spanish, verified English, machine-translated English, and a telegraphic rewrite. The rewrite drops determiners and copulas and substitutes abbreviations from the Grossman Liu et al. database. Scores across these renderings should not change. For sensitivity, we apply negation flips, laterality swaps and adjacent-region box shifts, which should each lower the score by one error. We measure, for every metric, the score spread across invariant renderings and the score drop per injected error. A good metric shows a small spread and a reliable drop.
The third experiment is diagnostic. We run one grounded generator in Spanish and in English on the same images and compare the typed errors. If degradation in Spanish appears mostly as wrong-location errors, that supports the visual-attention account of failure. If it appears mostly as wrong-concept or wrong-assertion errors, that supports the language-drift account behind MedVL-XLRepE. The literature currently leaves that dispute unresolved for generated reports.
The project succeeds if joint matching beats the decoupled composite on the multi-finding subset, and if its Spanish–English correlation gap is smaller than translate-then-GREEN's. It fails if both comparisons come out within noise.
References
- On the Automatic Generation of Medical Imaging Reports
- CUNI-a at ArchEHR-QA 2025: Do we need Giant LLMs for Clinical QA?
- RadGraph: Extracting Clinical Entities and Relations from Radiology Reports
- Evaluating progress in automatic chest X-ray radiology report generation
- GREEN: Generative Radiology Report Evaluation and Error Notation
- Clinically Grounded Agent-based Report Evaluation: An Interpretable Metric for Radiology Report Generation
- A multimodal multidomain multilingual medical foundation model for zero shot clinical diagnosis
- Knowledge Graph Embeddings for Multi-lingual Structured Representations of Radiology Reports
- PadChest-GR: A Bilingual Chest X-ray Dataset for Grounded Radiology Report Generation
- How Do Medical MLLMs Fail? A Study on Visual Grounding in Medical Images
- AnatomiX, an Anatomy-Aware Grounded Multimodal Large Language Model for Chest X-Ray Interpretation
- Parsing Clinical Text: How Good are the State-of-the-Art Parsers?
- A deep database of medical abbreviations and acronyms for natural language processing
- Developing healthcare language model embedding spaces
- A Visually-Grounded Parallel Corpus with Phrase-to-Region Linking
- Analyzing and Mitigating Cross-Lingual Degradation in Multilingual Medical VQA