Automatic radiology report generators write fluent text, but their clinical errors are hard to trace. When a model states "right-sided pneumothorax" and the reference says "no pneumothorax," nobody can tell whether the model misread the image or misworded a correct internal belief. Its internal state is an unstructured vector. Clinical evaluation metrics have their own structured vocabulary. RadGraph F1, the most widely adopted, parses each report into findings, anatomical locations, and a certainty flag (present, uncertain, absent), then scores the overlap. This proposal forces the generator to think in that same vocabulary. The model first fills a bottleneck: one slot per (finding, location) pair, each holding a discrete certainty state. It must then write the report only from that bottleneck. Each slot is identified by a text embedding of its description, such as "pleural effusion, left costophrenic angle," not by an integer index. Because of this, pairs never seen together in training still get a meaningful representation, and the slot set does not need to be enumerated or hand-curated.
Two things follow. First, every clinical error the metric reports can be assigned to a stage: perception (the bottleneck state is wrong) or verbalization (the text disagrees with the bottleneck). Second, negation stops being a subtle property of a sentence embedding and becomes an explicit, editable variable. The central bet is that a latent space expressed in the evaluation metric's units makes a report generator auditable and editable. It should also match the best reported clinical scores on MIMIC-CXR, because the structure removes a hallucination channel rather than adding capacity.
Core Claim
Borrow the interpretable-by-design decoder from transcriptomics and make two substitutions. The pathway mask becomes the RadGraph schema, and the fixed index per latent node becomes a semantic label embedding. The result is a report generator whose intermediate state can be scored directly with the same metric used on its output. Concretely, the claim has three testable parts. The bottleneck alone, read out as a set of triples, should score at least as high on RadGraph-style entity F1 as the triples parsed from the model's generated text. The text should almost never contradict its own bottleneck. And flipping a single slot's certainty should flip the corresponding sentence and nothing else. If all three hold and clinical scores stay near the DRRG level, the model's errors become attributable in a way no current system allows.
What Prior Work Does and Leaves Open
The mechanism comes from biology. In pathway-constrained autoencoders, each latent coordinate may reconstruct only the genes in its curated pathway. The decoder weights are multiplied by a binary membership mask, which forces that coordinate to act as a pathway activity score (Unsupervised Cell Functional Annotation for Single-Cell RNA-Seq). Such constraints cost little in prediction (Pathway Activity Score Learning for Dimensionality Reduction of Gene Expression Data) and track graded responses (Biologically Interpretable VAE with Supervision for Transcriptomics Data Under Ordinal Perturbations). We reuse the idea of a hard routing mask. The known weakness is that a static mask cannot express anything the database omits, and we address that below with a capacity-limited residual slot.
The schema comes from evaluation. RadGraph defines Anatomy and Observation entities with certainty labels and relations such as Located At. It reports parser micro F1 of 0.94 for entities and 0.82 for relations on MIMIC-CXR. Evaluating Progress in Automatic Chest X-Ray Radiology Report Generation showed that RadGraph F1 and the composite RadCliQ track radiologist error counts (Kendall tau-b 0.522 for RadCliQ). The same RadGraph paper also constrains the design: radiologists agree only moderately on fine-grained relations (Cohen's kappa 0.397). We therefore build slots from entities, certainty, and Located At only, and exclude the noisier Suggestive Of and Modify relations from the bottleneck.
Three systems are the closest neighbors, and each leaves a specific piece undone. DRRG uses RadGraph only to decide which tokens to mask during diffusion training. It conditions its decoder on a flat vector of 14 CheXpert probabilities while still letting the decoder see image features, so text can bypass the concepts. CBM-RAG passes 20 GPT-4-invented descriptors to an agent pipeline that shares no gradients with the concept stage and is judged by an LLM rather than a clinical graph metric. HiMed-RL and CheXalign optimize clinical scores as external rewards on a black-box model. CheXalign needed a length-controlled judge to stop the policy from gaming verbosity. None of these systems has an internal state that the metric can read.
Semantic label embedding supplies the slot representation. CARZero aligns text label descriptions to image patches with cross-attention and reaches AUC 0.810 on PadChest and 0.837 on its rare-disease subset. That result is evidence that description embeddings transfer to the long tail. However, CARZero classifies but does not generate, and its text queries leave negation implicit, which the lexical-metric literature warns against (Evaluating Medical Language Models Beyond Lexical Overlap). StructCBM shows that classifier-free matching against concept descriptions can be accurate and faithful, but its discriminative concepts grow quadratically with the number of classes. Graph Concept Bottleneck Models show that treating concepts as independent hurts. Our factorization into a finding embedding and a location embedding sidesteps the quadratic growth. We add slot-to-slot message passing only as an ablation, because Graph CBM's gain was modest (about 1 to 2 points).
Research Program
We first build the slot vocabulary without any manual annotation. The released RadGraph parser is run over MIMIC-CXR training reports. We collect every (observation, anatomy) pair joined by Located At, normalize surface forms, and keep pairs above a minimum frequency. Each report's supervision target then becomes, for every slot, one of four states: present, uncertain, absent, or not mentioned.
Each finding $o$ and location $a$ is embedded by a frozen biomedical text encoder $g$ applied to a short description. Slot scores are computed by cross-attention from the combined query to image patches, as in CARZero:
$$p(c \mid x, o, a) = \mathrm{softmax}_c\big(h_c(\mathrm{CrossAttn}(q_{o,a},\, V(x)))\big),\quad q_{o,a} = \phi\big(g(d_o), g(d_a)\big).$$
Here $x$ is the radiograph and $V(x)$ its patch features. The certainty state is $c$, and $d_o$ and $d_a$ are the text descriptions of the finding and location. The small combining network is $\phi$, and $h_c$ is a per-state output head. Because $q_{o,a}$ is built from the two text embeddings, any pair can be scored, including pairs absent from training.
The decoder is a small language model, around 0.6B parameters to match DRRG's scale. It attends only to the embeddings of slots whose state is not "not mentioned," each concatenated with a learned embedding of its certainty state. Image features are blocked from the decoder. This is the analogue of the masked decoder weights in the biology work.
To cope with an incomplete schema, we add a small pool of residual slots. These are free vectors the decoder may also attend to, with their total information capacity capped by a per-slot KL penalty. We report how much of the text they account for.
Training proceeds in two phases: first the bottleneck against the parsed targets, then the decoder with teacher-forced ground-truth slots. A final end-to-end phase feeds the decoder a straight-through relaxation of the predicted states. Beyond the model, the program delivers an evaluation protocol that decomposes RadGraph F1 into perception F1 and verbalization fidelity. This protocol applies to any future model with a readable intermediate state.
Baselines
The naive reference is the nearest-neighbor retrieval baseline of Baselines for Chest X-Ray Report Generation: return the training report whose image embedding is closest. That paper reports CheXpert macro F1 of 0.258 for 1-NN, above its CNN-RNN at 0.186. Evaluating Progress in Automatic Chest X-Ray Radiology Report Generation found that generative models improved only 5.5% over random retrieval under RadCliQ. We rerun 1-NN on our split with our image encoder, so any gain reflects structure rather than better features.
The strongest existing method is DRRG. It reports RadGraph-F1 0.281 and CheXpert-F1 0.549 on MIMIC-CXR, which are the numbers to match. It is the right competitor because it uses a decoder of similar size, already conditions on clinical concepts, and beat 7B and 14B models. We retrain it on our exact split and preprocessing with the same image backbone and tuning budget. If its reproduced numbers deviate from the reported ones, we report both. Beating DRRG means one of two outcomes. The strong version is matching its RadGraph-F1 within the seed-to-seed spread while also delivering bottleneck attribution and edit fidelity that DRRG cannot provide by construction. The clear win is exceeding its RadGraph-F1 and RadCliQ on held-out and rare-pair subsets.
A metric-optimizing black box is the third baseline: the same decoder trained with SFT and then GRPO against RadGraph F1 as a reward, following HiMed-RL. It gets the same number of reward calls as our end-to-end phase. CheXalign's reported 14% relative CheXbert gain came on an 8B model and a different protocol, so it is not directly comparable; we cite it only as evidence that metric-driven preference tuning works.
Four ablations isolate the key ingredients. The first replaces semantic slot embeddings with learned per-slot indices, which tests the label-embedding claim. The second shrinks the slot set to the 14 CheXpert observations, which yields a DRRG-style flat bottleneck. The third lets the decoder also see image patches, a leaky bottleneck that tests whether hard routing matters. The fourth removes the residual slots.
Why not just…?
Why not just condition a decoder on CheXpert labels, as DRRG does? Fourteen observations carry no location and cannot distinguish left from right, and DRRG's decoder can bypass them through image features. So its errors cannot be localized to perception or wording. The flat-bottleneck and leaky-bottleneck ablations test this directly. If both match our model on RadGraph-F1 and on edit fidelity, the richer vocabulary buys nothing and we would say so. We expect them to fail on laterality and location errors, which are precisely the relations RadGraph scores.
Why not just predict RadGraph triples with a classifier and have an LLM verbalize them? This two-stage pipeline, similar in spirit to CBM-RAG, is a real threat. It may well match our model on text quality, because end-to-end training may add little once the bottleneck is good. We treat it honestly as the stage-wise variant of our own model and train it with identical slots. If it ties, the contribution narrows to the metric-shaped, embedding-indexed bottleneck and the attribution protocol, which the pipeline inherits. The remaining question is verbalization fidelity: whether a prompted LLM adds unsupported findings. The contradiction rate against the bottleneck measures exactly this.
Why not just optimize RadGraph F1 as a reward on a black-box model? This optimizes the headline number but leaves nothing inside the model to inspect or edit. CheXalign's need for a length-controlled judge shows such optimization can be gamed. The GRPO baseline will show whether reward optimization alone matches our clinical scores. Even if it does, it cannot pass the slot-flip test, so the comparison separates "scores well" from "is controllable."
Why not just retrieve the nearest report? Retrieval produces fluent, human-written text and was surprisingly strong in Baselines for Chest X-Ray Report Generation. However, it can only return combinations of findings that already co-occurred in one training report. Our rare-combination split is designed to expose that limitation.
Proposed Experiments and Decisive Checks
We train and test on MIMIC-CXR with the standard report split and evaluate out of domain on IU X-Ray and PadChest-GR without fine-tuning. The implementation builds on ViLMedic and R2GenCMN. Metrics are RadGraph F1, CheXpert F1, and RadCliQ, plus BLEU as a fluency sanity check. The new decomposition adds perception F1, defined as the bottleneck triples scored against the parsed reference triples. It also adds verbalization fidelity, defined as RadGraph parsed from our output scored against our own bottleneck. Every result is reported over three seeds.
Four checks decide whether the claim survives. The first is an attribution check. Perception F1 plus the verbalization error rate should account for nearly all of the gap between our RadGraph F1 and a perfect score. We expect verbalization contradictions on the order of a few percent. If they are large, the bottleneck is not really governing the text.
The second is an intervention check. On a thousand test studies, we flip one slot at a time between present and absent and parse the output. The target triple should flip, and the rest of the parsed graph should stay unchanged. This operationalizes the causal-necessity concern raised against concept bottlenecks in StructCBM. The leaky-bottleneck ablation should fail this check.
The third is a held-out composition check. We withhold every training report containing a chosen set of about fifty (finding, location) pairs whose finding and location each appear elsewhere. We then measure slot AUC and RadGraph F1 on test studies containing them. Semantic slot embeddings should give well-above-chance AUC, while index-based slots cannot represent the pairs at all.
The fourth is a negation check. We build paired test queries that differ only in certainty and compare our explicit state variable against CARZero-style text-query similarity for separating present from absent. This directly tests the concern that description embeddings blur negation.
Failure is informative. If the full model trails DRRG by more than seed variance, then insisting on routing through RadGraph slots costs real accuracy. We would quantify that cost against the attribution and edit gains rather than hide it.
References
- Unsupervised Cell Functional Annotation for Single-Cell RNA-Seq
- Pathway Activity Score Learning for Dimensionality Reduction of Gene Expression Data
- Biologically Interpretable VAE with Supervision for Transcriptomics Data Under Ordinal Perturbations
- RadGraph: Extracting Clinical Entities and Relations from Radiology Reports
- Evaluating progress in automatic chest X-ray radiology report generation
- DRRG: A Discrete Diffusion Framework for Radiology Report Generation
- Towards Interpretable Radiology Report Generation via Concept Bottlenecks Using a Multi-agentic RAG
- Beyond N-grams: A Hierarchical Reward Learning Framework for Clinically-Aware Medical Report Generation
- CheXalign: Preference fine-tuning in chest X-ray interpretation models without human feedback
- CARZero: Cross-Attention Alignment for Radiology Zero-Shot Classification
- On the Automatic Generation of Medical Imaging Reports
- Adaptive Concept Discovery for Interpretable Few-Shot Text Classification
- Graph Concept Bottleneck Models
- Baselines for Chest X-Ray Report Generation