Language models that write radiology reports from chest X-rays often sound right while being clinically wrong. They drop a "no," invent a finding, or copy a sentence from a similar patient's report. Retrieval-augmented generation (RAG) means letting the model look up related material, here similar past reports, before it writes. Retrieval helps, but it adds a new failure: the model may paste findings from another patient. Reinforcement learning (RL) can train a model to retrieve well, but recent systems reward only final-answer string overlap. In radiology, that reward cannot tell a correct report from a dangerous one.
We propose to train a report-writing agent that decides when to retrieve, scoped to one anatomical topic at a time. The reward is computed claim by claim on a clinical graph of the report. For each atomic statement the model writes, a parser extracts its entities and relations: anatomy, findings, presence or absence. Each statement is then scored on three questions. Is it true for this patient according to the reference? Did a retrieved document actually contribute it? Was it copied from retrieved evidence without being true here? The agent earns credit when retrieval supplies true findings and pays a larger penalty when retrieval supplies false ones. A second, cheaper ingredient balances these objectives inside the RL update without the validation-heavy bandit controllers of earlier multi-reward work.
Core Claim
The obvious combination at this intersection is to take an RL-trained graph-retrieval agent and replace its token-F1 reward with a radiology graph metric. That plan has two problems. First, it is nearly an ablation of existing work. Second, it makes evaluation circular, because the training reward and the headline metric become the same thing. Our claim is narrower and more testable. The main value of a domain graph metric inside the RL loop is credit assignment for retrieval at the level of individual claims, not a better outcome score. A reward that separates "retrieval brought in a true finding" from "retrieval brought in a copied false finding" should reduce copy-hallucination and improve clinical correctness. We expect these gains to show up on metrics and human audits that the reward never sees. A whole-report graph reward, with or without retrieval, should not deliver the same gains.
Where This Sits
Three RL graph-retrieval systems define the method family. GraphRAG-R1 trains a multi-hop question-answering agent with group-relative policy optimization (GRPO, an RL method that scores each sample against others drawn for the same prompt). It adds a retrieval-attenuation reward and a cost-aware F1 reward under a fixed three-phase schedule. GTA-RAG shows that answer-only RL lets agents guess from memory while retrieving irrelevant documents. It fixes this with process rewards for retrieving the intended bridge documents, raising full-chain coverage from 54.3% to 74.1%, and its authors name specialized domains as open. Graph-R1 runs an agent over a hypergraph with format and answer rewards.
These papers supply our training machinery. GTA-RAG also supplies the motivation that retrieval needs its own credit signal. All three reward string overlap on open-domain QA, and their notion of "useful retrieval" is a document hit. We replace document hits with claim-level clinical graph support. We also add a penalty that only makes sense in generation: copying a true-elsewhere but false-here finding.
The radiology side supplies the metric and constrains the design. RadGraph parses reports into entity-relation graphs. Evaluating Progress in Automatic Chest X-Ray Radiology Report Generation builds RadGraph F1 and the composite RadCliQ on top of it, reaching Kendall-tau b of 0.522 with radiologist error counts. The same paper delivers a sobering result: leading systems improved only 5.5% by RadCliQ over a random-retrieval baseline. That result is why copying from retrieved reports must be measured explicitly.
FactMM-RAG uses RadGraph overlap to train a retriever of factually similar reports and prompts a generator with them. It reports gains of up to 6.5% in F1CheXbert and 2% in F1RadGraph. It has no policy learning and retrieves once per study. We reuse its retriever as the agent's search tool.
Rethinking Radiology Report Generation (LLaVA-TA) finds that fluent narrative is itself a source of hallucination. Replacing it with topic-guided atomic findings lifted RadGraph F1 from 29.4 to 44.0 and CheXpert F1-14 from 39.5 to 71.5 on MIMIC-CXR. We adopt this atomic output format, which is what makes claim-level scoring possible. We do not add fluency rewards, consistent with the separation of content from style argued in Style-Aware Radiology Report Generation.
On the clinical RL side, two systems optimize only one part of the pipeline. ReinRAG trains only the retriever over the medical concept graph UMLS. PathReasoner-R1 rewards correct knowledge-graph entities in reasoning steps but has no retrieval at inference. Its entity reward raised SlideBench-TCGA accuracy from 67.98% to 74.68%, which is evidence that entity-level rewards help where outcome supervision alone does not. Our agent learns both when to retrieve and what to write.
For balancing several rewards, DORB uses an Exp3 bandit across metrics. That bandit is our main balancing competitor, but it carries the validation overhead flagged in the research file.
Research Program
The agent is an open vision-language model in the LLaVA family, the family used by FactMM-RAG and LLaVA-TA. It writes a report as a sequence of anatomical topics: lungs, pleura, heart, mediastinum, bones, devices. For each topic it emits one or more atomic statements. Before any topic it may issue a retrieval call. The call combines the image with the topic name and returns the top-k training-set reports from the FactMM-RAG retriever, restricted to their sentences about that topic. Each retrieved sentence is already parsed into a RadGraph graph. Every generated statement must name the retrieved sentence it relies on, or the word "image" if it relies on none.
Rewards are computed after the full report. Let $g(c)$ be the RadGraph triples of statement $c$, let $G^*$ be the reference report's graph, and let $E(c)$ be the graph of the evidence that $c$ cites. Here $\mathrm{true}(c)$ means $g(c)$ matches $G^*$, including presence or absence of the finding. The claim reward is
$$ r(c) = \mathbb{1}[\mathrm{true}(c)]\,\big(1 + \lambda\,\mathbb{1}[g(c)\cap E(c)\neq\emptyset]\big) \;-\; \mu\,\mathbb{1}[\neg\mathrm{true}(c)]\,\big(1 + \mathbb{1}[g(c)\subseteq E(c)]\big). $$
Here $\lambda$ is the bonus for a true claim that retrieval supported, and $\mu$ is the base penalty for a false claim. That penalty doubles when the false claim is copied from evidence. Report-level terms are added on top: recall of reference findings never stated, a per-call retrieval cost in the spirit of GraphRAG-R1's cost-aware F1, and format validity. Recall and cost are reported as separate objectives rather than folded into one scalar in advance.
To balance the objectives, we normalize each one separately within each GRPO group of samples and only then sum the normalized advantages. No objective can then dominate simply because it has a larger scale or variance. The scheme costs nothing beyond the rollouts GRPO already draws.
Because RadGraph's extractor makes errors, the reward uses one extractor checkpoint and RadGraph-based evaluation uses a separately trained one. The primary claims rest on metrics the reward never touches: CheXbert-label F1, a negation-flip stress test, and radiologist error counts on a sample. RadCliQ includes RadGraph F1, so it is reported as partially in-loop.
Baselines
The naive reference point is the random-retrieval baseline from Evaluating Progress in Automatic Chest X-Ray Radiology Report Generation: return the report of a random or nearest training study. That paper shows current systems barely clear it by RadCliQ. We also run the base vision-language model fine-tuned on MIMIC-CXR reports with ordinary supervised learning and no retrieval. Any trained system that fails to beat both by a clear margin on CheXbert F1 has shown nothing.
The strongest non-RL retrieval competitor is FactMM-RAG: its RadGraph-trained retriever, single-shot retrieval, and a generator fine-tuned on the same data. The file reports only relative gains, so no absolute FactMM-RAG score on MIMIC-CXR was found on Lacuna, and we will reproduce it ourselves.
The strongest reported absolute numbers on MIMIC-CXR in the file belong to LLaVA-TA: RadGraph F1 of 44.0 and CheXpert F1-14 of 71.5. Because we adopt its atomic format, LLaVA-TA trained on our backbone with supervised learning is the baseline that isolates everything RL and retrieval add. Beating it means exceeding 71.5 CheXpert F1-14 under the same backbone and data, or exceeding our reproduction of it if that falls short of the reported figure. The gain must hold under the held-out RadGraph extractor.
Three ablations isolate the key ingredient. The first is GRPO with whole-report RadGraph F1 as the only reward and no retrieval. The second is the same reward with retrieval allowed, which is the "obvious combination." The third is our claim reward with the copy-penalty multiplier removed. For balancing, we compare per-objective normalization against a static weighted sum and against a DORB Exp3 bandit over the same objectives. DORB is charged its validation passes in the compute budget.
All systems share the backbone, the MIMIC-CXR training split, and one retrieval corpus. Every RL variant gets the same rollout count and the same hyperparameter-search budget of eight configurations, selected on validation CheXbert F1, not RadGraph F1.
Why not just…?
Why not just run RL on RadGraph F1 directly, without retrieval? This is the most serious threat. PathReasoner-R1 shows entity-level rewards help without inference-time retrieval. If the no-retrieval ablation matches us on CheXbert F1 and radiologist error counts, then retrieval is not earning its cost, and we will say so. Our prediction is narrower: retrieval helps on rare findings, where the image signal is weak and similar cases carry information. We will therefore break results down by finding frequency. The whole-report reward variant also tests the hacking concern that a single metric invites, which has been documented for single factuality rewards in Tailoring Self-Rationalizers with Multi-Reward Distillation.
Why not just use FactMM-RAG's retriever with supervised fine-tuning? Supervised training on retrieved context gives the model no signal that separates using evidence from copying it. The random-retrieval result in Evaluating Progress suggests copying is exactly where gains stall. The copy-hallucination rate is the decisive measurement: the fraction of false statements whose triples appear in retrieved evidence. If supervised FactMM-RAG already has a low rate, our main motivation weakens.
Why not just adopt LLaVA-TA's topic-guided format and stop there? Its reported gains are large and come without RL. Our method is only worth its complexity if it adds measurable correctness on top of that format, which is why LLaVA-TA on our backbone is a required baseline rather than an afterthought.
Why not just use a DORB-style bandit to balance the rewards? DORB adapts to rewards that change over training, but it needs frequent validation evaluations. Running the RadGraph parser makes those evaluations expensive. We run DORB directly and compare at matched wall-clock compute. If it wins at that budget, we adopt it.
Proposed Experiments
Training and in-distribution testing use MIMIC-CXR. Out-of-institution transfer uses IU-Xray. Robustness checks use MIMIC-ABN and Two-view CXR, where MLRG reports a 2.7% F1 RadGraph gain as a reference point.
The first decisive check targets the core claim: whether claim-level retrieval credit lowers the copy-hallucination rate and raises CheXbert F1 relative to the obvious-combination ablation, with RadGraph F1 under the held-out extractor moving the same way.
The second is a negation-flip stress set. We take reference statements, flip their presence or absence, and check whether each system's generated statements track the image rather than the phrasing of retrieved evidence. This targets the failure that lexical metrics miss, as described in Evaluating Medical Language Models Beyond Lexical Overlap.
The third is a blinded audit. Radiologists count clinically significant errors in 200 reports per system. Expert raters are required because non-experts miss dangerous omissions in fluent text. We test whether the ranking of systems by error count matches their ranking by in-loop metrics. A mismatch would indicate reward hacking.
The fourth is retrieval behavior: calls per report, broken down by finding rarity, alongside the per-objective normalization versus bandit versus static-sum comparison at matched compute. The proposal succeeds if the full system beats LLaVA-TA on our backbone on CheXbert F1 and radiologist error counts. It also needs a lower copy-hallucination rate than both FactMM-RAG and the whole-report RL ablation. It fails if the no-retrieval RL ablation matches it on all of these.
References
- GraphRAG-R1: Graph Retrieval-Augmented Generation with Process-Constrained Reinforcement Learning
- GTA-RAG: Graph-Trajectory-Augmented Reinforcement Learning for Multi-Turn Retrieval-Augmented Reasoning
- Graph-R1: Towards Agentic GraphRAG Framework via End-to-end Reinforcement Learning
- RadGraph: Extracting Clinical Entities and Relations from Radiology Reports
- Evaluating progress in automatic chest X-ray radiology report generation
- Fact-Aware Multimodal Retrieval Augmentation for Accurate Medical Radiology Report Generation
- Rethinking Radiology Report Generation: From Narrative Flow to Topic-Guided Findings
- Style-Aware Radiology Report Generation with RadGraph and Few-Shot Prompting
- Leaps Beyond the Seen: Reinforced Reasoning Augmented Generation for Clinical Notes
- PathReasoner-R1: Instilling Structured Reasoning into Pathology Vision-Language Model via Knowledge-Guided Policy Optimization
- DORB: Dynamically Optimizing Multiple Rewards with Bandits
- Tailoring Self-Rationalizers with Multi-Reward Distillation
- Enhanced Contrastive Learning with Multi-view Longitudinal Data for Chest X-ray Report Generation
- CUNI-a at ArchEHR-QA 2025: Do we need Giant LLMs for Clinical QA?