When a model that writes chest X-ray reports is trained at one hospital and used at another, we want to know how its mistakes change: does it invent findings, miss findings, put them on the wrong side of the chest, or misjudge their severity? Answering that question needs an automated grader that counts each kind of error. The graders now trusted for this, such as GREEN and RadGraph-F1, were validated almost entirely on reports from US academic centers. Their reliability on another hospital's writing style, templates, or language has never been checked. If a grader misses fabrications more often on one hospital's reports, a measured "cross-site hallucination profile" partly reflects the grader rather than the model.
This proposal treats the grader as a measuring instrument and calibrates it separately at each hospital before using it. We insert known errors of each type into real reports from four institutions. That lets us estimate, for each site and each error type, how often the grader catches a real error and how often it flags a clean report. We then use those rates to correct the error counts for models trained at a single center and tested elsewhere. The central question is whether conclusions about cross-site hallucination change once the grader's own site bias is removed. Two kinds of conclusion are at stake: which models generalize best, and whether the shift mainly causes fabrications or omissions. If they do change, every multi-center report-generation evaluation that skips this step is suspect.
Core Claim
Existing work splits into two halves that have not been joined. One half builds error-type graders for radiology reports and checks them against radiologists, always inside one distribution. Evaluating Progress in Automatic Chest X-Ray Radiology Report Generation built a six-category clinical error taxonomy and the RadCliQ composite score on MIMIC-CXR. GREEN distilled GPT-4 error notation into small local models and validated them on the 200-pair ReXVal set. ReEvalMed injected omissions, fabrications and inaccuracies into 70 US reports to test whether metrics respond correctly to error severity. The other half measures cross-hospital degradation with coarse classification scores and no error taxonomy. Pooch et al. reported an AUC drop from 0.83 to 0.68 when moving from NIH to Stanford. They also found that apparent transfer between MIMIC-CXR and CheXpert is inflated because both datasets were labeled by the same rule-based tool.
That labeler finding is what motivates this proposal. If a shared labeling tool can manufacture apparent generalization in classification, a shared grader can manufacture apparent hallucination profiles in report generation. Our claim is testable. Error-type graders have detection rates that vary by site and by error type. Correcting for this variation changes substantive conclusions about cross-site report generation. And corrected counts agree with radiologists' counts better than raw counts do.
The closest existing work in method is ReEvalMed's injection design, and we reuse its technique. It differs from us in two ways. It measures whether a metric moves in the right direction, while we estimate per-site detection and false-flag rates and use them to correct downstream counts. It also works on one US distribution, while we work across sites. The closest work in scope is HEAL-MedVQA, which mixes US and Vietnamese images. It probes question-answering shortcuts rather than narrative reports, and it reports no single-center-trained, external-site-tested splits.
Research Program
The study uses four report-bearing sites. MIMIC-CXR (Boston) is the training site for all models. CheXpert Plus (Stanford) shares MIMIC's labeling lineage. OpenI (Indiana) has a different reporting style. PadChest (Alicante) has Spanish reports. VinDr-CXR has no narrative reports, so it is excluded from the text arm. It is used only as an image-only stress test, through classifier-extracted findings.
We note one deliberate confound up front. GREEN's training data included MIMIC-CXR, PadChest and OpenI reports but not CheXpert Plus. CheXpert Plus is therefore the cleanest held-out site for the grader, while PadChest and OpenI are "seen-style" sites. This contrast is useful in itself: if grader bias appears mainly at the unseen site, the bias comes from the grader's training data.
The first stage is a site-stratified injection set. From each site we sample about 150 reference reports, stratified by frontal view type because CheXmask found AP views systematically harder than PA views. Into each report we inject one error drawn from the shared taxonomy: fabricated finding, omitted finding, laterality or location swap, severity change, or negation reversal. We also create clean paraphrase controls that should not be flagged.
Injections are written by templated edits over RadGraph entities. They are then checked by one radiologist per site for plausibility, and by a second radiologist on a 20% overlap, because RadGraph reports relation agreement as low as kappa 0.397 on CheXpert. PadChest is handled in two versions, native Spanish and machine-translated English. Translation is itself an intervention whose effect we measure rather than assume away.
From this set we estimate, for each site $c$ and error type $k$, two quantities. The detection rate $s_{c,k}$ is the fraction of injected type-$k$ errors the grader flags. The false-flag rate $f_{c,k}$ is the rate at which it flags type $k$ in clean controls. When the grader reports $n^{\text{obs}}_{c,k}$ errors of type $k$ across $m_c$ generated reports at site $c$, the corrected count is the standard misclassification adjustment:
$$\hat n_{c,k} = \frac{n^{\text{obs}}_{c,k} - f_{c,k}\, m_c}{s_{c,k}}$$
Confidence intervals come from bootstrapping over injected reports.
The second stage trains report generators on MIMIC-CXR only and evaluates them at all four sites with a single locked pipeline: fixed preprocessing, fixed decoding, fixed grader versions and fixed seeds. The pipeline is locked because NerfBaselines and BARS both show that unlocked pipelines produce rankings that dissolve once controlled. The generators are an image-only retrieval baseline, a Pragmatic LLaMA-style model conditioned on the ordering indication and on classifier-predicted findings, and the 0.6B discrete diffusion generator DRRG.
We then report per-site error profiles under raw and corrected counts. To split image-side from text-side causes, we compare the Pragmatic model with and without the indication field. An indication written in a different site's ordering style is a known source of site-specific behavior; Chen et al. show local practice patterns failing to transfer in a different clinical task.
The third stage is validation. At each site, a radiologist annotates about 100 real model outputs using the taxonomy. This sample is large enough to test whether corrected counts are closer to human counts than raw counts are. It is far smaller than full expert annotation of every output, which is exactly what the injection set is meant to avoid.
Baselines
The naive reference point is reporting BLEU and ROUGE-L per site, which is still common practice. GREEN reports Kendall's tau of 0.35 (BLEU) and 0.56 (ROUGE-L) against expert error counts on ReXVal. These metrics run in the same locked pipeline and offer no error-type breakdown, so they serve only as a floor.
The strongest existing grader is GREEN itself, run uncorrected, using the released RadLLaMA-2 and RadPhi-2 checkpoints. It reports Kendall's tau of 0.63–0.64 with radiologist error counts on ReXVal and matches radiologist preference 62–68% of the time. It is the strongest competitor because it already produces per-type error counts and was trained on several sites' reports. If any grader is already site-neutral, it is GREEN. We run it with identical prompts, inputs and hardware at every site, and the corrected version reuses exactly the same outputs, so the only difference is the adjustment step.
The second grader is RadCliQ from Yu et al., with Kendall tau-b of 0.522 against radiologist scores on MIMIC-CXR. It is included because it depends on RadGraph entity extraction trained on MIMIC-CXR and CheXpert, which makes it a natural candidate for Pooch-style shared-pipeline bias.
We also report the distribution-aware CRG Score per site. Its only reported results are on CT-RATE (e.g., CT-CHAT 0.368 versus a trivial-model floor of 0.33). We found no reported result on Lacuna for CRG Score on chest X-ray report sites, so here it serves as a check against "always normal" degenerate behavior, which may differ by site prevalence.
The key ablation isolates the site-specific part of the correction. We compare three versions: correcting with rates pooled across all sites, correcting with MIMIC-derived rates applied everywhere, and correcting with each site's own rates. If pooled or MIMIC-only rates perform as well as site-specific ones, then the grader's bias is uniform, and the "site" part of the method adds nothing.
"Beating" the strongest baseline is defined as follows. At sites outside GREEN's training data, site-corrected GREEN counts must show a higher Kendall's tau with radiologist counts than raw GREEN counts, with non-overlapping bootstrap intervals. At least one per-type conclusion must also reverse or change significantly: either the ordering of fabrication versus omission increase under shift, or the cross-site ranking of the three generators.
Why not just…?
Why not just run GREEN at every site and read off the per-type counts? This is the main threat, and it may be correct. GREEN saw PadChest and OpenI reports in training, so its rates there might already be stable. The injection set measures exactly this. If detection and false-flag rates are flat across sites within their confidence intervals, and the ablation shows pooled correction matches site-specific correction, we will report that GREEN is site-neutral. That is a useful negative result, and it licenses everyone else's multi-site use of GREEN. The proposal then reduces to its second stage, a site-resolved hallucination profile, which is still new according to the gap map in the research file. We expect the unseen CheXpert Plus site and the Spanish-native PadChest condition to be where flatness fails, but we do not assume it.
Why not just have radiologists annotate everything at each site? Expert annotation is the gold standard, and we use it to validate. The obstacle is scale and disagreement. RadGraph's relation agreement of 0.397 on CheXpert means every site needs double annotation, and ReEvalMed's 400 pairs from one distribution already took substantial expert effort. The injection set costs plausibility checks rather than full error hunting. The 100-output validation per site tests whether the cheaper instrument gives the right answer.
Why not just ask GPT-4 to judge reports at each site? GREEN reports that zero-shot GPT-4 preference matched radiologists only 23% of the time. Evaluating Evaluation Metrics found GPT-4 the best hallucination detector on general benchmarks (weighted F1 about 0.74), but it also found that detectors' behavior shifts unpredictably with the evaluated model. Liu et al. showed GPT-4's own knowledge varies sharply by country. An LLM judge would need exactly the same site calibration, so we include it as a fourth grader inside the injection protocol rather than as a replacement for it.
Why not just translate PadChest into English and treat it as another site? Translation can remove or add hedging, negation and laterality cues, which are the error types that matter here. By running native and translated versions through the same injection set, we measure how much translation changes the grader's detection rates. We do not presume it is neutral.
Proposed Experiments and Decisive Checks
The first decisive check is whether detection rates $s_{c,k}$ differ significantly across sites for any error type under GREEN and RadCliQ. We test this with a site-by-error-type interaction in a logistic model over injected reports. We expect negation and laterality to be the most site-sensitive, because these are the cues most tied to local templates.
The second check is whether corrected counts track radiologist counts better than raw counts at each external site. We measure this by Kendall's tau and by absolute count error on the validation outputs.
The third check is whether correction changes a substantive conclusion. This means either the ranking of the retrieval, Pragmatic and DRRG generators at external sites, or the direction of the fabrication-versus-omission shift relative to MIMIC. We also check whether adding the indication field moves errors from fabrication toward omission at sites whose ordering style differs from MIMIC's.
As a secondary analysis in the spirit of Liu et al.'s finding that coarse aggregation hides local failure, we report how much a pooled four-site average conceals per-site, per-type variation. If none of the first three checks fires, the honest conclusion is that current graders are portable across these sites, and the program's value lies in the locked, site-resolved error profiles it publishes.
References
- Evaluating progress in automatic chest X-ray radiology report generation
- GREEN: Generative Radiology Report Evaluation and Error Notation
- ReEvalMed: Rethinking Medical Report Evaluation by Aligning Metrics with Real-World Clinical Judgment
- Can We Trust Deep Learning Based Diagnosis? The Impact of Domain Shift in Chest Radiograph Classification
- Localizing Before Answering: A Hallucination Evaluation Benchmark for Grounded Medical Multimodal LLMs
- CheXmask: a large-scale dataset of anatomical segmentation masks for multi-center chest x-ray images
- RadGraph: Extracting Clinical Entities and Relations from Radiology Reports
- NerfBaselines: Consistent and Reproducible Evaluation of Novel View Synthesis Methods
- BARS: Towards Open Benchmarking for Recommender Systems
- Pragmatic Radiology Report Generation
- DRRG: A Discrete Diffusion Framework for Radiology Report Generation
- Correlation of broad-spectrum antibiotic utilization patterns with invasive candidiasis in critically ill patients and development of an early prediction model
- CRG Score: A Distribution-Aware Clinical Metric for Radiology Report Generation
- Evaluating Evaluation Metrics – The Mirage of Hallucination Detection
- Operationalizing Geographic Diversity for the Evaluation of AI-Generated Content