Vision-language models that write chest X-ray reports hallucinate in a specific way. They assert findings the image does not support, often because a correlated finding was already written ("cardiomegaly" pulls in "pulmonary edema"). Once that false claim is on the page, the rest of the report tends to agree with it. The best current safety net, conformal claim filtering, works after the fact: it splits the finished report into atomic claims, deletes the ones it doubts, and asks another LLM to stitch the survivors back together. This note proposes to move that decision inside generation. We write the structured report format as a formal grammar and enforce it token by token (grammar-constrained decoding, which masks any next token the grammar forbids). The grammar makes every clinical claim appear as a predictable slot: an anatomical section, a finding term, then a status word. At each status slot we measure how much the model's choice actually depends on the image, and a threshold calibrated on held-out data decides whether the model may commit to "present"/"absent" or must write "indeterminate."
The payoff is a report that stays structurally intact and never needs a second re-writing model. Its definite claims carry a finite-sample bound on error rate, and uncertainty shows up as an explicit, visible word rather than a silent deletion. Because the decision happens before later tokens are written, a doubtful early claim cannot seed a chain of confirming errors. The grammar's role here is not to fix formatting, which fine-tuning largely already handles. Its role is to expose where the claims are, so a statistical guarantee can be applied at the moment of commitment.
Core Claim
A structured-report grammar turns open-ended generation into a sequence of enumerable decision points. At those points, a visual-reliance score, calibrated with conformal risk control, can bound the error rate of definite clinical claims during decoding. We expect this to match post-hoc conformal filtering on precision while giving fewer silent omissions, less error propagation between correlated findings, and no re-synthesis step. We also claim the ordering the grammar imposes matters. Forcing the status word before any descriptive text, the "premature commitment" failure noted in the grammar-decoding literature, should measurably hurt accuracy. A description-first grammar should recover it.
Relation to Prior Work
Three papers motivate the problem. CWCD splits chest X-ray reporting into 8 anatomical categories with per-category adapters. It contrasts outputs on images with and without masked regions to suppress tokens driven by language priors. It documents that attention to image tokens decays as the report grows. CWCD, however, decodes each section as unconstrained text, needs bounding boxes from LATTE-CXR, and offers no error guarantee. Towards Statistical Factuality Guarantee for Large Vision-Language Models (ConfLVLM) supplies the statistical tool: split conformal prediction over atomic claims. Its post-hoc design is what we change. It deletes claims after generation, which raises omissions, and it needs an external LLM to rewrite the survivors. KCTS motivates intervening during decoding, since it shows unsupported tokens snowball. Its tree search is costly and text-only, and our intervention is a single extra forward pass at a small number of slots.
The method rests on two lines of work. Automated Structured Radiology Report Generation supplies the SRRG-Findings format (8 organ-system headers, bulleted findings), SRR-BERT with its 55-finding taxonomy, and the F1-SRR-BERT metric. That work enforces structure only through fine-tuning. Flexible and Efficient Grammar-Constrained Decoding (GREATGRAMMA) and GRAMMAR-LLM supply the decoding engines. GREATGRAMMA reports 5–32 ms per-token masking, and GRAMMAR-LLM reports 100% validity. Both have been tested only on code, JSON and text classification. GRAMMAR-LLM explicitly concedes that grammatical validity cannot stop "valid but incorrect" early choices, and that concession is exactly the gap this proposal fills.
Other work constrains the design. The "structured output reasoning tax" and premature-commitment concerns summarized in the file warn that masking can push models into low-probability paths. We therefore treat grammar ordering as an experimental variable, not a fixed choice. Energy-Guided Decoding and Octopus show that decoding-time contrast reduces object hallucination on general benchmarks. They are competitors for the scoring function, not for the guarantee.
Research Program
The first component is the grammar. Each of the 8 SRRG headers expands to zero or more bullets. The grammar has two orderings. In the status-first ordering, a bullet is a finding term drawn from the SRR-BERT taxonomy, then a status token from {present, absent, indeterminate}, then free-text qualifiers. In the description-first ordering, a free-text clause comes first, followed by an explicit "status:" slot. The finding-term position is a finite vocabulary, and the qualifiers are free text inside the grammar. We compile both grammars with GREATGRAMMA against the tokenizer of the backbone. The backbone is the LLaVA-Rad-style model used as CWCD's strongest baseline, fine-tuned on SRRG-Findings.
The second component is the visual-reliance score at each status slot. Let $x$ be the image, $\tilde{x}$ an image-ablated input (the image replaced by a blurred or noised copy, which needs no bounding boxes), $h$ the prefix generated so far, and $y$ the model's top status token. The score is
$$s(y) = \log p(y \mid x, h) - \log p(y \mid \tilde{x}, h),$$
which is large when the choice is driven by the image and small when the language prior alone would produce it. This is where the cross-modal gap enters operationally. We do not try to close the gap geometrically. We measure, per claim, whether evidence crossed it. We also compare against the raw status probability and a CWCD-style anatomy-masked variant where boxes exist.
The third component is calibration. On a held-out calibration split, we label each generated claim as correct or incorrect using SRR-BERT labels on the reference structured report. We then choose a threshold $\hat{\tau}$ by conformal risk control so that
$$\mathbb{E}\left[\frac{\#\{\text{incorrect definite claims}\}}{\max(1,\#\{\text{definite claims}\})}\right] \le \lambda,$$
where $\lambda$ is the user's tolerated error rate. At test time, any slot with $s(y) < \hat{\tau}$ is forced to "indeterminate" by the grammar mask. Because later slots depend on earlier forced choices, we calibrate the whole decoding procedure at the report level, with reports treated as exchangeable. The guarantee then covers the actual sequential process, not isolated claims. An honest caveat: the guarantee is relative to SRR-BERT labels, which have F1 0.836 at the finest granularity. A radiologist-audited subset would be needed to state clinical validity.
Baselines
The naive reference point is the SRRG-fine-tuned backbone decoded greedily with no grammar and no filtering. It tells us the starting claim error rate and how often fine-tuning alone already yields a parseable report. The strongest published generator on this data is CWCD, which reports overall F1 62.51, F1RadGraph 24.60 and BLEU-4 6.60 on SRRG-Findings, against LLaVA-Rad's overall F1 of 62.12 (CWCD). We will rerun CWCD on the same splits rather than copy numbers, because our metrics include claim-level error that CWCD did not report.
The strongest competitor on the actual question, bounded claim error, is ConfLVLM applied post-hoc to the same backbone's free-text output. It uses the same calibration split, the same $\lambda$, and both its internal-probability scorer and its BiomedCLIP scorer. Its reported headline result is a claim error reduction from 87.8% to 10.0% for LLaVA-1.5 on MSCOCO with a 95.3% true positive rate. It was evaluated on 500 MIMIC-CXR images, but no reported result was found on Lacuna for its MIMIC-CXR error rate. All methods share the backbone, the calibration data and the decoding budget. The comparison to CWCD and to our slot-level method counts added forward passes explicitly.
Beating ConfLVLM means that, at matched realized error among definite claims, our method has a lower rate of silently missing abnormal findings. It must also show higher F1-SRR-BERT on the definite claims and no re-synthesis hallucinations. Three ablations isolate the ingredients. The first uses the grammar without calibration, which tests whether structure alone helps. The second keeps calibration but drops the grammar, applying the threshold to claims extracted post-hoc; this is effectively ConfLVLM with our score. The third swaps the visual-reliance score for raw status probability.
Why not just…?
Why not just fine-tune on SRRG and skip the grammar? For formatting alone, this is probably enough. We expect the naive baseline's parse rate to be high, and we will report it rather than claim a validity win. The grammar earns its place only if exposing slots during decoding matters. The ablation that applies the same calibrated threshold to claims extracted after free generation tests exactly this. If it matches the in-generation method on omissions and on error propagation between correlated findings, the grammar is unnecessary and we will say so.
Why not just run ConfLVLM post-hoc? This is the real threat. Post-hoc filtering already achieves large error reductions on MSCOCO. Our argument is that deleting a claim after the fact cannot undo its influence on later claims, as KCTS argues for text. Deletion also hides uncertainty from the reader, and re-synthesis adds a second model that can hallucinate. The decisive measurement is conditional error. On pairs of correlated findings (for example cardiomegaly and edema), we measure how often a wrong first claim is followed by a wrong second claim under each method at matched $\lambda$. If post-hoc filtering shows no excess conditional error and similar omission, the in-generation design adds little.
Why not just use contrastive decoding everywhere, as in CWCD or Octopus? Contrastive decoding lowers hallucination on average but gives no bound, and its strength is a tuned hyperparameter. We include CWCD directly. We also check whether our calibrated method reaches a lower realized error at equal coverage, and whether it holds its target error when CWCD's does not.
Why not threshold the model's own status probability? Softmax confidence is known to be overconfident. The ablation replacing $s(y)$ with raw probability tests directly whether the image-ablation contrast buys better selectivity, meaning fewer indeterminates at the same error.
Proposed Experiments
We train and calibrate on SRRG-Findings splits of MIMIC-CXR and evaluate in-distribution. Out-of-distribution evaluation uses IU X-Ray and the HOPPR test set of 1,300 radiographs across 8 states. The HOPPR test is where exchangeability should break, and we will report how far realized error exceeds $\lambda$ under shift instead of hiding it. The metrics are realized claim error among definite claims across a sweep of $\lambda$, the indeterminate rate, and the omission rate for abnormal findings present in the reference. We also report F1-SRR-BERT, F1RadGraph, RadFact precision and recall (MAIRA-2), the conditional-error measure for correlated finding pairs, and wall-clock latency including GREATGRAMMA masking and the extra ablated forward passes at status slots.
The ordering experiment compares status-first with description-first grammars at identical calibration. It directly measures the premature-commitment cost that the grammar-decoding literature warns about.
Two outcomes would each settle part of the claim. First, if in-generation calibration holds realized error at or below $\lambda$ in-distribution with lower abnormal-finding omission and lower conditional error than post-hoc ConfLVLM at the same $\lambda$, the core claim stands. If it does not, the grammar-exposed slots add nothing beyond post-hoc filtering. Second, if the visual-reliance score yields fewer indeterminates than raw probability at equal error, the cross-modal contrast is doing real work rather than acting as a relabeled confidence score.
References
- CWCD: Category-Wise Contrastive Decoding for Structured Medical Report Generation
- Towards Statistical Factuality Guarantee for Large Vision-Language Models
- KCTS: Knowledge-Constrained Tree Search Decoding with Token-Level Hallucination Detection
- Automated Structured Radiology Report Generation
- Flexible and Efficient Grammar-Constrained Decoding
- GRAMMAR-LLM: Grammar-Constrained Natural Language Generation
- Energy-Guided Decoding for Object Hallucination Mitigation
- Octopus: Alleviating Hallucination via Dynamic Contrastive Decoding
- MAIRA-2: Grounded Radiology Report Generation