AI systems that write chest X-ray reports are usually trained at one hospital and scored with automated clinical metrics. The most common of these is RadGraph F1. It parses each report into a small graph of findings and anatomy, then counts how much of that graph the generated report gets right. A natural next step is to report this score separately for each demographic group and penalize the spread between groups. Recent fairness benchmarks already do this for classification and segmentation, under the name "equity-scaled" scores. The catch is that the parser behind RadGraph F1 is itself a model trained on one hospital's reports. It is known to perform worse on reports from other hospitals. If it also performs worse for some patient groups at those hospitals, an equity-scaled score could report a fairness gap the report generator never caused, or hide one it did.
This note proposes to measure that confound directly. We would take report generators trained at one hospital, run them at external hospitals, and score them per demographic group with automated clinical metrics. We would also obtain radiologist-annotated ground truth on a stratified sample. This lets us split each group's measured score into two parts: what the generator got wrong, and what the scoring tool mis-scored. The deliverable is an equity-scaled clinical score corrected for evaluator error. It comes with a finding about whether "disparities widen under hospital transfer" is true for report generation or partly an artifact of single-hospital metrics.
Core Claim
The claim is that measured demographic gaps in clinical report quality across hospitals are a sum of generator error and evaluator error. Both are inherited from single-center training, so they cannot be read off a group-wise metric without a correction. The motivation comes from two existing results. FairDomain studies fundus-image segmentation and classification under a shift between two imaging devices. It shows that the shift nearly triples the race disparity in optic cup segmentation. RadGraph builds the entity–relation parser and shows its accuracy drops when moving from MIMIC-CXR reports to Stanford's CheXpert reports, because each hospital writes differently. No one has asked whether that second drop is uneven across patient groups. If it is, then applying an equity-scaled score to RadGraph F1 across hospitals is unsafe. The obvious pairing of these two results, an equity-scaled RadGraph F1, is therefore unsafe as an evaluation tool until the parser's own per-group error is measured.
Relation to Prior Work
Equity-scaled scoring comes from two sources. FairSeg and FairDomain define equity-scaled Dice and AUC, which divide a score by a factor that grows with group disparity. FMBench subtracts a variance penalty from average performance and applies it to multimodal model outputs on a single ophthalmology dataset. We reuse their scoring rules. What changes is the quantity being scaled. Their scaled quantities, AUC, Dice, and BLEU, have reliable ground truth or deterministic scoring, so evaluator error never arose for them. For a learned clinical parser applied at a new hospital, evaluator error is the central problem.
Maximum Weighted Loss Discrepancy justifies variance as a smooth stand-in for worst-group gap. It also implies that noise in per-group scores inflates measured variance, which is exactly the effect we correct for.
Domain-specific report metrics also come from two sources. RadGraph supplies the parser. Evaluating Progress in Automatic Chest X-Ray Radiology Report Generation builds RadCliQ, a combination of BLEU and RadGraph F1 that reaches Kendall-tau b of 0.522 with radiologist error counts. Both papers validate their metrics only as population averages, mostly at one hospital. The RadGraph paper also reports a constraint we must respect: radiologists agree only moderately on relations, with Cohen's kappa of 0.397 on CheXpert. Our ground truth will therefore be noisy as well, and the design must use multiple annotators and treat agreement as a ceiling.
Several report generators have been tested across hospitals. LLaVA-TA splits reports into topic-guided findings grounded in anatomical masks and reports transfer to IU-Xray. Style-Aware generation predicts a RadGraph first and verbalizes it with a prompted LLM. All of them report aggregate scores only. Separately from the main question, LLaVA-TA's design gives us a testable side hypothesis: generators that lean less on text priors might widen group gaps less under transfer.
Research Program
The work has three parts.
The first part is a data audit and cohort assembly. MIMIC-CXR (MIMIC-CXR) is the source hospital. CheXpert, IU-Xray, and the English side of PadChest-GR are the external hospitals. The research file does not confirm which demographic fields each external dataset releases. Step zero is therefore to catalog the available attributes, likely age and sex at minimum, with race or language where released. Any site lacking demographic fields will be used only for aggregate transfer, not for equity claims. Intersectional groups will be reported only where cell sizes support confidence intervals.
The second part is the measurement-error decomposition. For a clinical metric $M$, let $\hat M_{g,s}$ be the automated score for group $g$ at site $s$, and let $M^{*}_{g,s}$ be the score computed with radiologist-annotated graphs on both the generated and the reference report. The evaluator bias is $b_{g,s} = \hat M_{g,s} - M^{*}_{g,s}$. We estimate $b_{g,s}$ on a stratified annotated sample of a few hundred studies per site-by-group cell, each annotated by two or more radiologists. We then extend it to the full test set with a calibration model that predicts per-report evaluator error from report features such as length, negation density, and section structure. The corrected equity-scaled score is
$$\text{ES-}M = \frac{\bar M}{1 + \sum_{g} \lvert \bar M - (\hat M_{g} - \hat b_{g}) \rvert},$$
where $\bar M$ is the corrected population mean and $\hat b_g$ is the estimated evaluator bias for group $g$. This follows the inverse-disparity form of the equity-scaled scores in FairSeg. We report bootstrap intervals that propagate uncertainty in $\hat b_g$.
The third part is a style control. Documentation dialect is a plausible reason the parser could fail unevenly across groups. For example, a hospital's reports for patients with complex histories may use different phrasing. To test this, we rewrite external reference reports into a MIMIC-like style. Following Style-Aware generation, the rewrite is conditioned on their human-annotated graphs. We then re-run the parser. If evaluator bias shrinks after style normalization, dialect is the mechanism. We verify a sample of rewrites with radiologists to confirm that clinical content is unchanged.
Baselines
The naive reference point is aggregate RadGraph F1 at each external site with no group breakdown. This is current practice, as in LLaVA-TA, and it cannot detect disparity at all.
The second baseline is FMBench-style fairness-aware performance computed over BLEU and over GREEN or LLM-as-judge scores (FMBench). It is the strongest existing equity metric for generated reports.
The third baseline is the uncorrected equity-scaled RadGraph F1 and equity-scaled RadCliQ. These are the obvious combination and the most direct competitor. RadCliQ is the strongest existing single clinical metric, with a reported Kendall-tau b of 0.522 against radiologists on MIMIC-CXR (Evaluating Progress). No reported result was found on Lacuna for RadCliQ or RadGraph F1 broken down by demographic group, or for RadCliQ at external hospitals.
All metrics are computed on identical generated reports, identical test splits, and identical group definitions. LLM-judge prompts are fixed in advance rather than tuned on the annotated sample. The annotated sample is split so that the calibration model is never scored on the studies it was fit on.
Our score beats these baselines if two things hold on held-out annotated cells. First, its per-group gaps rank-correlate with radiologist-derived gaps $M^{*}_{g,s}$ significantly better than the uncorrected scores, with a Kendall-tau improvement whose bootstrap interval excludes zero. Second, it changes at least one substantive conclusion, such as which group is worst-served or whether disparity widens from source to target.
One ablation removes the correction, which is the uncorrected baseline. A second removes the calibration model and uses only raw per-cell bias estimates, which isolates whether the extension to unannotated reports adds value.
The generators scored are those with the strongest MIMIC-trained results in the file. LLaVA-TA reports RadGraph F1 of 44.0 and CheXpert F1-14 of 71.5 on MIMIC-CXR, versus 29.4 and 39.5 for LLaVA-Rad. Its IU-Xray result is described only qualitatively, with no reported number found on Lacuna. The other generators are the Style-Aware two-step model (RadGraph F1 0.228 versus 0.186 on MIMIC-CXR), MLRG (Enhanced Contrastive Learning), and LLM-CXR. All use released or retrained checkpoints at equal compute, with no target-site tuning.
Why not just…?
Why not just compute group-wise RadGraph F1 and report the gap? This is the real threat. If the parser's error turns out to be roughly equal across groups at each site, the correction cancels in the gap and this simpler audit is enough. The decomposition experiment decides the question directly. Even in that outcome, the paper delivers the first validated cross-hospital equity audit of report generators, plus evidence that the shortcut is safe, which no one can currently claim.
Why not just use an LLM judge or GREEN instead of a learned parser? LLM judges are also trained mostly on dominant documentation styles, and nothing in the file shows they score evenly across groups or hospitals. We treat them as a competing evaluator and estimate their bias $b_{g,s}$ against the same radiologist annotations. If the LLM judge shows near-zero group bias, we will recommend it over the corrected parser.
Why not just fine-tune the parser on each target hospital? This fixes the ruler rather than measuring it, and it needs target-site annotations of the same kind we already collect. We include it as an arm: fine-tune on half the annotated pool and test whether residual group bias remains. Retraining could reduce the average error drop seen in RadGraph while leaving uneven error intact. That is precisely the case our correction addresses.
Why not just use bigger generators? FMBench found that scaling from 7B to 26B did not reduce group variance and sometimes increased it. In any case, scale changes the model being measured, not the bias of the measuring tool.
Decisive Checks
The central test is whether $b_{g,s}$ differs across groups at any external site, beyond the noise floor set by inter-radiologist agreement. If it does not, the hypothesis is refuted. If it does, the follow-up is whether the source-to-target disparity widening that the uncorrected score reports survives correction. A secondary test is whether topic-guided generation (LLaVA-TA) widens corrected gaps less than narrative generators do, which would extend the narrative-bias argument to equity. A final check is whether style normalization of references removes most of $b_{g,s}$, which would identify documentation dialect as the channel through which single-center training contaminates fairness measurement.
References
- FairDomain: Achieving Fairness in Cross-Domain Medical Image Segmentation and Classification
- RadGraph: Extracting Clinical Entities and Relations from Radiology Reports
- FairSeg: A Large-Scale Medical Image Segmentation Dataset for Fairness Learning Using Segment Anything Model with Fair Error-Bound Scaling
- FMBench: Benchmarking Fairness in Multimodal Large Language Models on Medical Tasks
- Maximum Weighted Loss Discrepancy.
- Evaluating progress in automatic chest X-ray radiology report generation
- Rethinking Radiology Report Generation: From Narrative Flow to Topic-Guided Findings
- Style-Aware Radiology Report Generation with RadGraph and Few-Shot Prompting
- MIMIC-CXR: A large publicly available database of labeled chest radiographs
- PadChest-GR: A Bilingual Chest X-ray Dataset for Grounded Radiology Report Generation
- Enhanced Contrastive Learning with Multi-view Longitudinal Data for Chest X-ray Report Generation
- LLM-CXR: Instruction-Finetuned LLM for CXR Image Understanding and Generation