Chest X-ray report generators have to learn two things: how to see a finding and how to write it down. When expert-structured reports are scarce, the writing half is usually trained from very few examples, or it is borrowed from a language model that already writes fluently. That fluency causes the problem. An autoregressive decoder (a network that writes a sequence one token at a time, each token conditioned on the ones before it) quickly learns that "the lungs are clear" is usually followed by "the heart is normal." It then stops consulting the image. We propose to pre-train the whole image-to-text network, encoder, decoder, and the cross-attention layers that let the decoder query the image, on unlabeled radiographs alone. The target sequences are built so that the only way to predict the next token is to look at a specific region of the image.
The idea borrows the "pseudo-language" trick from speech, where decoders are pre-trained to transcribe audio into discrete codes discovered by clustering. We change it in one essential way. Each pseudo-token is tied to a spatial query ("what is at region k?"), the queries arrive in random order, and earlier answers are partially hidden. The decoder can therefore learn to route attention to the image, but it cannot learn a narrative. After pre-training, the same query mechanism is reused: region queries are replaced by the anatomical section headers of a structured report, rather than being thrown away. We test whether this gives data-efficient structured report generation that also stays grounded in the image.
Core Claim
Joint encoder-decoder pre-training is known to rescue sequence generation in low-label regimes. In speech, a pre-trained encoder paired with a randomly initialized decoder exceeds 100% word error rate on 10 hours of LibriSpeech, while jointly pre-trained Wav2Seq reaches 38.1% (Wav2Seq). In semantic parsing, a pre-trained BART decoder carries structural priors worth roughly 10x in labeled data (Low-Resource Domain Adaptation for Compositional Task-Oriented Semantic Parsing).
In radiology, the same autoregressive priors are the documented source of hallucination. Rethinking Radiology Report Generation shows that, as generated text accumulates, model outputs on real images converge to outputs on black images. Our claim is that the benefit and the harm come from what the decoder is pre-trained to predict, not from pre-training the decoder at all. A pretext task whose targets are image-determined, spatially keyed, and order-free should keep the data-efficiency benefit and remove the narrative shortcut.
The direct test is this. With 1% of structured reports, the model should match or beat a decoder pre-trained on free-text reports on clinical-accuracy metrics. It should also degrade sharply, not gracefully, when the image is replaced by a black one.
Relation to Prior Work
Three lines of work motivate this proposal.
Speech seq2seq pre-training supplies the method. Wav2Seq clusters HuBERT features and applies byte-pair encoding to form pseudo-subwords. Joint Encoder-Decoder Self-Supervised Pre-training for ASR trains a decoder on collapsed cluster-ID sequences and reports up to 25.8% relative improvement on the 100h split. Both operate on one-dimensional audio, where temporal order is the natural target order. Both also discard the pseudo-vocabulary at fine-tuning, which leaves the cross-attention layers to adapt to a brand-new output space. We move the idea to two-dimensional images, where no natural order exists. We deliberately randomize the order, and we preserve the query interface across the switch from pre-training to fine-tuning.
Label-efficient radiograph pre-training supplies the encoder baseline and marks the gap. Masked Record Modeling jointly reconstructs masked image patches and masked report tokens, reaching 88.5% mean AUC on CheXpert with 1% labels. However, its text side is non-autoregressive masked-token prediction, and it conditions on a single averaged image vector. It never trains cross-attention from generated tokens to image patches, and it is never evaluated on generation. DeCon jointly pre-trains a dense convolutional decoder with contrastive losses, but that decoder is a pixel head, not a sequence generator.
Structured report generation constrains the design. Style-Aware Radiology Report Generation shows that generating structured RadGraph facts first raises RadGraph F1 from 0.186 to 0.228, but its image-to-graph model is purely supervised. Automated Structured Radiology Report Generation provides the SRRG-Findings data (184k reports organized by anatomical system) and the F1-SRR-BERT metric, and evaluates only supervised fine-tuning. LLaVA-TA (Rethinking Radiology Report Generation) gets grounding by decomposing reports into topic-guided findings. To do so it needs an external segmentation tool, offline LLM decomposition, and a large pre-trained multimodal model.
The parsing result is the apparent contradiction we address head on. BART's priors helped there because tree structure is the true regularity of the task. Radiology narratives, by contrast, contain regularities that are clinically false shortcuts. We keep structural priors (sections, slots, query-to-region routing) and exclude sentence-level priors.
Research Program
The pre-training task works as follows. A vision transformer encoder receives a corrupted radiograph, either masked or downsampled, following MRM's finding that super-resolution corruption forces fine detail. A frozen teacher encoder, for example the MRM or a masked-autoencoder image encoder, processes the clean image. Its patch features are pooled over a multi-scale grid of $K$ regions. Each pooled feature is assigned to one of $V$ k-means clusters, giving a code $c_k$ for region $k$.
The decoder receives learned region-query embeddings $q_k$ in a random permutation $\pi$. It is trained with
$$ \mathcal{L} = -\sum_{t=1}^{K} \log p\big(c_{\pi(t)} \mid q_{\pi(t)},\, \tilde{c}_{\pi(<t)},\, E(\tilde{x})\big), $$
where $E(\tilde{x})$ is the encoder output on the corrupted image and $\tilde{c}_{\pi(<t)}$ are the previously emitted codes with a fraction $\rho$ replaced by a mask token. Random order removes any fixed "narrative" sequence to memorize. Masking earlier codes limits the decoder's ability to infer a region from its correlated neighbors. Together they push predictive information through cross-attention.
Pre-training uses images only: MIMIC-CXR images with their reports withheld, plus CheXpert and NIH ChestX-ray images. Using no reports means the comparison against report-based pre-training isolates the objective rather than the data.
For fine-tuning, the decoder is trained on SRRG-Findings subsets of 1%, 10%, and 100%. The structured output is serialized so that each anatomical section header (for example "Lungs and Airways") acts as a query token, followed by that section's findings text. The decoder body and cross-attention are kept. Only the output vocabulary and header embeddings are new. We also compare initializing each header from the mean of the region queries it covers against random initialization, which directly measures the vocabulary-swap cost that Wav2Seq leaves open.
The analysis has three parts. The Blank Image Test is run per section and per token position. Cross-attention mass is measured on MS-CXR phrase boxes. Rare-finding recall is examined using SRR-BERT's 55-class taxonomy, since the file notes pre-training has not closed the long-tail gap.
Baselines
The naive reference is the MRM image encoder with a randomly initialized transformer decoder, fine-tuned on the same SRRG subsets. This is the "just attach a decoder" design that fails catastrophically in speech. If it is not clearly worse than ours at 1%, the premise of decoder pre-training for images is wrong.
The strongest competitor on our specific question is narrative joint pre-training. The architecture, encoder initialization, and compute are identical, but the model is pre-trained autoregressively on the roughly 212k free-text MIMIC-CXR image-report pairs before fine-tuning on the SRRG subsets. It is the most natural way to pre-train a report decoder, and it uses real language rather than pseudo-codes. We expect it to win on BLEU and lose on grounding.
We also run a disconnected variant: the MRM encoder plus a decoder pre-trained on report text alone. This tests whether a text-only prior suffices.
For reference points, not compute-matched ones, we cite published numbers on MIMIC-CXR. LLaVA-TA reports RadGraph F1 of 44.0 and CheXpert F1-14 of 71.5, against 29.4 and 39.5 for LLaVA-Rad (Rethinking Radiology Report Generation). The Style-Aware two-step model reports RadGraph F1 of 0.228 (Style-Aware). CheXpert-Plus reports BLEU 14.84 on SRRG-Impression (Automated SRRG). We found no reported F1-SRR-BERT result on SRRG-Findings in Lacuna for any model, so our full-data supervised run establishes that number. LLaVA-TA uses a far larger language model and full supervision, so we would treat approaching its RadGraph F1 at 10% labels as a strong result rather than a promised one.
All in-house baselines share the same pre-training image budget, GPU-hours, fine-tuning subsets, and a fixed hyperparameter search of equal size. Beating the narrative baseline means two things together: equal or higher F1-SRR-BERT and CheXpert F1-14 at 1% and 10%, and a statistically larger Blank-Image gap.
The ablations isolate the key ingredient. We compare fixed raster order against random order, $\rho=0$ against $\rho>0$, and dropping region queries (unkeyed code sequences in the Wav2Seq style) against keeping them. Teacher choice (MRM versus a plain masked autoencoder) tests whether the gains come from the objective or from the teacher's features.
Why not just…?
Why not just pre-train on the free-text reports MIMIC already provides? This is the real threat, because reports are plentiful and carry true clinical vocabulary. The evidence that it falls short on grounding is the Blank Image Test in Rethinking Radiology Report Generation, but that result concerns fine-tuned large vision-language models, not a compute-matched pre-training comparison. Our narrative baseline settles it directly. If it matches us on clinical F1 and on the Blank-Image gap, the pseudo-language objective adds nothing. In that case we would test the obvious hybrid, keyed pseudo-codes followed by report pre-training, as the fallback contribution.
Why not just predict the 55 SRR-BERT classes with a classifier and fill a template? A multi-label head on an MRM encoder is simple, fast, and cannot hallucinate narrative. It is a serious competitor on classification-style metrics, so we include it and score its templated output with the same metrics. It cannot express location, severity, or free-text qualifiers outside the taxonomy, so we expect it to lose on RadGraph F1. If it ties us there as well, generation is not earning its cost in this regime.
Why not just take the MRM encoder and a randomly initialized decoder? In speech this collapses at 10h (Wav2Seq). On parsing it trails pre-trained seq2seq by 8.7% on nested queries (Low-Resource Domain Adaptation). Images with a strong encoder may behave differently, which is why this is our naive baseline rather than an assumption.
Why not just prompt a frozen LLM with extracted findings? The cascade cannot recover findings the extractor misses, and it raises privacy and cost concerns (Style-Aware). It also does not address how to train the extractor with few labels, which is the problem we target.
Proposed Experiments and Decisive Checks
The primary experiment fine-tunes all systems on 1%, 10%, and 100% of SRRG-Findings. Each is evaluated on the MIMIC-CXR test split with F1-SRR-BERT, RadGraph F1, and CheXpert F1-14, reporting BLEU only as a fluency diagnostic, since it rewards narrative phrasing (Automated SRRG).
Grounding is measured two ways. For the Blank Image Test, we report, per section, the divergence between outputs on real and black images as a function of prefix length. For grounding accuracy, we report the MS-CXR hit score of cross-attention.
Robustness is tested on out-of-distribution data. IU-Xray and the 1,300-sample HOPPR set test whether keyed pre-training transfers across hospitals. BLEU is reported to collapse on HOPPR while clinical metrics hold (Automated SRRG).
The proposal succeeds only if two conditions hold together. At 1% labels, keyed pseudo-language pre-training must beat the random-decoder baseline and at least tie the narrative-pre-trained baseline on F1-SRR-BERT. Its Blank-Image divergence must also stay flat across prefix length where the narrative baseline's decays.
If the random-order and code-masking ablation performs no better than fixed-order pre-training, then the order-free design is not the source of any gain. We would report that honestly as a negative result.
References
- Wav2Seq: Pre-Training Speech-to-Text Encoder-Decoder Models Using Pseudo Languages
- Low-Resource Domain Adaptation for Compositional Task-Oriented Semantic Parsing
- Rethinking Radiology Report Generation: From Narrative Flow to Topic-Guided Findings
- Joint Encoder-Decoder Self-Supervised Pre-training for ASR
- Advancing Radiograph Representation Learning with Masked Record Modeling
- Beyond the Encoder: Joint Encoder-Decoder Contrastive Pre-Training Improves Dense Prediction
- Style-Aware Radiology Report Generation with RadGraph and Few-Shot Prompting
- Automated Structured Radiology Report Generation