Radiologists spend much of their day turning what they see on a scan into a written report. Systems that draft these reports automatically could remove that bottleneck, but only if a radiologist can trust which sentences in the draft to accept without checking. One reliable way to earn that trust is conformal prediction. It is a statistical wrapper around any model that uses a held-out set of labeled examples to pick an acceptance threshold. The threshold comes with a guarantee: the error rate among accepted outputs stays below a chosen level. The catch is that the guarantee holds only on average over the whole population.
Chest X-ray studies are not a homogeneous population. Some arrive with a lateral view, a prior study, and a stated clinical question. Others are single portable films from the ICU with no history at all. An average guarantee can hide a filter that passes too many wrong sentences on the incomplete studies and throws away too many correct ones on the complete studies. We propose a claim filter whose error guarantee holds separately for each pattern of missing inputs. Inside each pattern, it borrows statistical strength across similar findings using the routing signals of a small mixture-of-experts verifier, which is a network that sends each input to one of several specialized sub-networks. The accepted claims become the auto-accepted part of the draft. Everything else is flagged for the radiologist.
Core Claim
The claim has three parts. First, in chest X-ray report generation, which inputs are missing is a major and observable driver of how often generated claims are wrong. Second, calibrating the acceptance threshold within each missingness pattern restores error control for that pattern, and this holds regardless of why the inputs are missing. Third, the remaining unevenness, which is error concentrated on rare findings within a pattern, can be reduced without giving up validity. The method weights calibration examples by how similarly a missingness-aware expert verifier routes them.
The second part deserves emphasis. The mask is the record of which views and history are present. If calibration and test studies are drawn from the same source, then conditioning on the observed mask keeps the examples interchangeable within each mask, whatever the missingness mechanism. The MCAR assumption (missingness completely at random) that limits Conformal Prediction with Missing Values is needed only for their "nested" variant, which artificially hides features. It is not needed for exact per-mask calibration. This matters in radiology because missingness there is clearly not random: portable single-view films go to sicker patients.
Position Relative to Prior Work
Four papers each supply one piece, and none combines them.
Towards Statistical Factuality Guarantee for Large Vision-Language Models (ConfLVLM) supplies the claim-level mechanism. It splits a generated report into atomic claims, scores each claim with an internal or external critic, and sets a conformal threshold that bounds the claim error rate. It was evaluated on 500 MIMIC-CXR studies. Its calibration is marginal and assumes complete inputs. We keep its decomposition and change the conditioning.
Conformal Language Modeling gives a related sentence-level guarantee on MIMIC-CXR using CheXbert as the judge of correctness. It also treats the model as a black box and ignores input incompleteness. Both papers motivate our work, since each leaves a clear gap.
Conformal Prediction with Missing Values defines coverage that holds for every missingness mask. It shows that ordinary conformal intervals undercover the subgroups missing informative features, and proposes exact per-mask and nested-augmentation calibration. Its evidence comes from scalar tabular regression, including TraumaBase. We transfer its conditioning target to claim-level risk on generated text. We also point out that its MCAR restriction does not bind for exact stratification.
Adaptive Conformal Prediction via Mixture-of-Experts Gating Similarity (MoE-CP) weights calibration points by the similarity of their gating vectors, meaning each point's probability distribution over experts. A randomization step preserves the overall guarantee. That paper uses fully observed tabular inputs. We reuse its weighting, but only inside a mask stratum, and compute the gating from a verifier that knows which inputs are absent.
For the verifier itself, Flex-MoE supplies the architecture. It uses a learnable embedding bank indexed by which modalities are present, plus experts specialized to specific modality combinations. It beat the best dense baseline by 7.6% accuracy on ADNI under missing modalities. However, it produces uncalibrated classifications, so it gives us the architecture but no guarantee.
On the generation side, MLRG handles missing history with learned absence tokens. Its authors explicitly call for uncertainty estimates to flag cases needing human review. Pragmatic Radiology Report Generation cuts hallucinations by 69.5% by conditioning on the clinical indication. These papers constrain the design in a specific way. Improving Radiology Report Generation Systems by Removing Hallucinated References to Non-existent Priors finds that 76.3% of MIMIC-CXR reports reference a prior study. Comparative claims are therefore common, and they cannot be verified when no prior exists. A filter that is blind to missingness will handle them badly, because it has no way to know that the prior needed to check them is absent.
Research Program
The system has three parts. The generator can be any existing drafter; we use MLRG and an R2GenGPT-style frozen-LLM model (R2GenGPT). This keeps the guarantee separate from any particular generator.
The verifier is a small sparse mixture of experts. It reads the available image features, the prior study if one is present, the indication if one is present, and a single candidate claim. Following Flex-MoE, it replaces each absent input with a learned embedding indexed by that mask. It outputs a score for how conformant the claim is.
The calibration layer works as follows. For a test study with mask $m$, it collects calibration studies with the same mask. It weights each calibration claim by gating similarity to the test claim, using the MoE-CP randomized weighting. It then picks the loosest threshold $\hat\tau_m$ such that the weighted error among accepted claims satisfies
$$\Pr\big(\text{err}_m(\hat\tau_m) \le \lambda\big) \ge 1-\alpha,$$
where $\text{err}_m$ is the fraction of accepted claims that are false among studies with mask $m$, $\lambda$ is the tolerated error rate, and $\alpha$ is the allowed failure probability. The analysis gives a guarantee that holds exactly for each mask. We also prove, or bound empirically, the degree of finding-level adaptivity the weighting adds.
Masks are defined by three observable indicators: lateral view present, prior study present, and indication present. That gives eight patterns. Correctness labels for claims come from matching against the reference report with CheXbert labels and RadGraph entities, following Conformal Language Modeling. Because this labeling is automatic and noisy, we will audit a sample by hand and report how label noise affects the achieved error rates.
A second analysis separates missing information from patient acuity. Among studies that do have a lateral view, we hide it and re-run the generator and verifier. If error rises only on the genuinely single-view studies but not on the artificially hidden ones, acuity is the driver. In that case nested augmentation in the style of CP-MDA-Nested would calibrate against the wrong population.
Baselines
The naive reference point is the unfiltered draft from each generator, along with a raw token-probability threshold tuned to the target error on the pooled calibration set. Neither has any conditional guarantee. Together they show how much error a radiologist inherits with no filter. Act or Defer shows why this reference point matters: for pancreatic cancer on EHRSHOT, standard thresholding collapsed to about 10% selected coverage, while stratified conformal selection stayed above 99%.
The strongest existing method is ConfLVLM with marginal calibration, using the same generator, the same claim decomposition, and the same pool of calibration studies. It is the only published system that bounds claim error for radiology report generation from images. Its reported numbers in the file are for MSCOCO: claim error fell from 87.8% to 10.0% with a 95.3% true-positive rate for removing wrong claims. For MIMIC-CXR, no reported claim-error result was found on Lacuna. Conformal Language Modeling is a second strong competitor, but no reported MIMIC-CXR number was found on Lacuna for it either. Both will be re-run under our protocol.
We add two adaptations from the missing-data literature. The first is exact per-mask calibration (CP-MDA-Exact), which stratifies by mask with no gating weights. The second is nested augmentation (CP-MDA-Nested), which calibrates by hiding views. Both use our verifier's scores, so they differ from our method only in how the threshold is set.
Two ablations isolate the key ingredient. In the first, the MoE verifier is replaced by a dense verifier with the same parameter count that receives the mask as an input token. In the second, gating similarity is replaced by cosine similarity of the dense verifier's penultimate embeddings, a generic localized-conformal weighting.
Every threshold method uses the same calibration studies and the same test split. All verifiers get the same training data and the same hyperparameter search budget.
Beating the strongest baseline means three things at the same $\lambda$ and $\alpha$. The worst-mask error must stay at or below $\lambda$ where marginal ConfLVLM exceeds it. The fraction of true claims retained must be at least as high on complete studies. And the worst error across mask-by-finding cells must be lower than exact per-mask calibration achieves, without retaining fewer claims.
Why not just…?
Why not just stratify by mask and stop there? With only eight patterns and a large dataset, exact per-mask calibration will probably deliver mask-level validity on its own. We treat it as part of the method, not as a competitor. The open question is the finer level. Errors on uncommon findings, such as pneumothorax on single-view portable films with no prior, are hidden inside a mask average. Stratifying further by finding starves the calibration cells of data. If per-mask-and-finding calibration with a fallback to coarser cells matches our worst-cell error and retention, the gating component adds nothing. We would then report the simpler method as the answer.
Why not just use a dense verifier with a mask token and generic embedding similarity? This is the main threat to the mixture-of-experts ingredient. MoE-CP argues that gating vectors group inputs by how the model processes them, and Flex-MoE shows that specialized experts help under missing modalities. Neither result has been tested on claim verification, however. The two ablations settle this directly. If the dense verifier with embedding weighting matches us, we drop the mixture of experts and keep the per-mask risk control.
Why not just build a better generator, such as MLRG or Pragmatic LLaMA, and skip filtering? Better generators reduce errors but give no bound on the errors that remain. MLRG's authors themselves ask for uncertainty estimates to route cases to humans. Filtering and generation are complementary, and we run our filter on both generators to show it adds value on top of either.
Why not just use softmax confidence or Monte Carlo dropout? These scores carry no finite-sample guarantee. The EHRSHOT collapse to roughly 10% coverage reported in Act or Defer is direct evidence that raw thresholds fail on rare, high-stakes groups. The token-probability baseline tests this on our data.
Proposed Experiments
We use MIMIC-CXR with its longitudinal structure, Two-view CXR, and MIMIC-ABN to stress abnormal findings. All three are used by MLRG. We split by patient into training data for the verifier, calibration studies, and test studies. We sweep $\lambda$ from 5% to 20% at $\alpha = 0.1$.
The first decisive check is per-mask error. Marginal ConfLVLM should exceed $\lambda$ on the portable single-view, no-prior strata, while our method stays at or below it.
The second check simulates a hospital with a different case mix by reweighting the test set so single-view ICU studies dominate. Marginal guarantees should break under this shift. Per-mask guarantees should hold, because the shift changes only the proportions of each mask.
The third check targets normal bias. A filter can meet its error target by quietly deleting abnormal claims and keeping easy "no acute findings" sentences. The review of generation methods (Automatic radiology report generation with deep learning) flags this failure mode. We therefore report positive-finding recall among accepted claims, and the fraction of claims flagged for review, as the proxy for radiologist workload.
The fourth check is the view-hiding analysis. It decides whether nested augmentation is even the right comparison for radiology.
The proposal succeeds if per-mask risk control holds where marginal control fails. It also requires that gating-weighted calibration lower the worst finding-level error relative to both exact stratification and generic embedding weighting. If it does not, we will say so and publish the simpler per-mask filter.
References
- Conformal Prediction with Missing Values
- Towards Statistical Factuality Guarantee for Large Vision-Language Models
- Conformal Language Modeling
- Adaptive Conformal Prediction via Mixture-of-Experts Gating Similarity
- Flex-MoE: Modeling Arbitrary Modality Combination via the Flexible Mixture-of-Experts
- Enhanced Contrastive Learning with Multi-view Longitudinal Data for Chest X-ray Report Generation
- Pragmatic Radiology Report Generation
- Improving Radiology Report Generation Systems by Removing Hallucinated References to Non-existent Priors
- R2GenGPT: Radiology Report Generation with frozen LLMs
- Act or Defer: Error-Controlled Decision Policies for EHR Foundation Models
- Automatic radiology report generation with deep learning: a comprehensive review of methods and advances