A common way to make a text generator "see" an image is to compress the image into one feature vector and use it as the generator's starting memory, before it writes the first word. This is visual state initialization: the vector seeds the recurrent hidden state or becomes a learned prefix token. It works within one dataset, but it fails badly under a specific kind of shift. In radiology report summarization, QIAI at MEDIQA 2021 seeded a GRU encoder and decoder with DenseNet121 chest X-ray features. Trained on MIMIC-CXR and tested on Indiana University X-rays, the multimodal model did worse than a model that ignored the image entirely. It recovered only when trained on data mixed across both institutions. A seed the decoder trusts from the very first step is exactly where a wrong prior does the most damage.
We propose that this collapse is caused by what goes into the seed, not by seeding as a technique. A raw image embedding carries scanner, protocol and site signatures alongside the clinical content. A domain-structured seed carries only quantities a clinician would read off the image, such as a vector of calibrated finding probabilities per anatomical region, and should be far less site-specific. This follows the lesson of LuminaECG, which recast electrocardiograms as calibrated, clinically delineated primitives rather than generic pixels and then generalized across US, German and Chinese cohorts. We test whether forcing the visual pathway of a report generator through such a bottleneck of clinical primitives turns visual seeding from a liability under institution shift into a gain. We also test whether the size of the out-of-distribution penalty can be predicted from how easily the seed reveals which hospital it came from.
Core Claim
The claim has three parts. First, under train-on-one-site, test-on-another evaluation, seeding a report generator with a low-dimensional vector of domain primitives will beat both text-only generation and raw-feature seeding. Raw-feature seeding will fall below text-only, reproducing the QIAI failure.
Second, the out-of-distribution penalty of any seed will track how much site identity it carries. We measure that as the accuracy of a linear probe that predicts the source institution from the seed vector.
Third, once the content is fixed to primitives, the injection channel matters much less than the content. The channels are a hidden-state seed, a prefix token, or plain-text serialization of the findings.
The third part is deliberately falsifiable against the simplest alternative, described below. If text serialization of the findings does as well as state seeding, we will report that. The contribution then becomes the diagnosis: site-specific nuisance content in the seed causes the collapse, and the probe predicts it. We would not then claim that a new injection mechanism is the remedy.
Relation to Prior Work
Our mechanism comes from the visual-initialization line. Incorporating Global Visual Features into Attention-based NMT compared mapping VGG19 features into encoder or decoder hidden states against treating the image as an input word, and found the initialization variants stronger on Multi30k. Multi-modal gated recurrent units for image description argued that seeding only at the first step limits distraction from repeated visual input. Multimodal Grounding for Sequence-to-Sequence Speech Recognition compared start-token replacement with hidden-state initialization. All of these studies use generic ImageNet-style features and evaluate on a single distribution. We keep their injection channels as arms of the experiment and change only the content of the seed.
The closest precedent for changing seed content is Improving Image Captioning by Leveraging Knowledge Graphs (CNet-NIC). It concatenated ConceptNet concept embeddings with Inception features to initialize an LSTM captioner on MS-COCO, reaching 107.2 CIDEr-D. An ablation in that paper found that seeding with concept embeddings alone beat image-only seeding. That ablation motivates our hypothesis. Our setting differs in three ways. CNet-NIC used open-domain commonsense, a single distribution, and symbolic knowledge from an external graph. We use expert primitives read from the image itself, cross-institution shift, and an explicit test of whether removing nuisance content explains the robustness. CNet-NIC also documents a constraint we inherit: errors in the upstream detector cascade into the seed. Our primitive extractor can fail the same way, so we measure its calibration under shift directly.
The domain-knowledge line supplies the motivation and the method for building primitives. LuminaECG shows that a 2B vision-language model grounded in clinical measurement primitives outperformed far larger generalist models on critical findings across cohorts. LuminaECG grounds the model through rendered inputs and prompting, not through the generator's initial state, and it does not isolate why primitives transfer. We test that question.
Towards Medical Machine Reading Comprehension with Structural Knowledge and Plain Text (KMQA) injects knowledge-graph triples as pseudo-sentences. It finds that 44% of residual errors come from missing entities and 32% from noisy retrieval, which is a warning that external symbolic knowledge is brittle. This is one reason we derive primitives from the image rather than retrieve them. TheoremQA acts as a caution in the other direction. Simply supplying domain rules as text did not help models apply them. Domain content in the context window is therefore not automatically used. This is why we test the injection channel instead of assuming text serialization suffices.
Research Program
The core system reproduces QIAI's setting as closely as the file allows. A bidirectional GRU encodes the "findings" text and a GRU decoder writes the "impression."
The visual pathway has three possible seed contents. The first is the raw 1024-dimensional DenseNet121 vector, as in QIAI. The second is a primitive vector $p$: calibrated probabilities for a fixed set of radiological findings, each tied to an anatomical region. These come from a finding classifier trained only on MIMIC-CXR images, with labels extracted from MIMIC training reports. The third is the concatenation of the two.
The primitive seed is a learned linear map $h_0 = W p + b$, where $h_0$ is the initial hidden state and $W, b$ are trained with the generator. Because $p$ is defined per study rather than per image, frontal and lateral views can be combined by pooling their finding probabilities. This addresses QIAI's single-view restriction without changing the generator.
Each content is crossed with three channels: hidden-state seeding of the encoder and decoder (QIAI's encdecinit), a single prefix token, and serialization of the thresholded findings as a short text preamble. To check that the result is not specific to recurrent networks, the same grid is repeated with a small pretrained transformer decoder, where hidden-state seeding is replaced by a prefix.
The analysis component is the site probe. For every trained model we freeze the seed vectors on held-out MIMIC and Indiana studies and train a linear classifier to predict institution. Across the full grid of contents and channels, we regress the out-of-distribution drop in report quality on probe accuracy.
Baselines
The naive reference point is the text-only generator with no visual input. QIAI found this was the configuration to beat out of distribution. On their mixed-institution Indiana-dev split, the monomodal baseline reports BLEU 26.93 and ROUGE-2 F1 52.18 (QIAI). Any visual seed that fails to beat text-only under MIMIC-only training has failed the basic test.
The strongest existing method in this exact setting is QIAI's allv ensemble. It combines hidden-state initialization with multiplicative fusion into the encoder annotations and target embeddings. On mix-split Indiana-dev it reports BLEU 28.32 and ROUGE-2 F1 54.38, significant at $p \le 0.05$. There is no reported result found on Lacuna for QIAI-style models trained on MIMIC-CXR alone and scored numerically on Indiana. We will therefore rerun allv and encdecinit ourselves under MIMIC-only training, using the same architecture sizes, the same six-model ensembling, and the same hyperparameter search budget per arm. The mix-split allv model serves as an oracle upper bound that has seen target-site data, which our method never does. Beating the strongest existing method means two things. Our primitive seed, trained on MIMIC only, must exceed both MIMIC-only allv and text-only on Indiana. It must also close at least half the gap to mix-split allv.
The modern competitor is a compact vision-language model fine-tuned with LoRA on MIMIC, attending to image patch tokens throughout generation. This is the approach LuminaECG uses for ECGs, and the comparison asks whether continuous visual access beats a narrow seed. It receives the same training data and a matched compute budget.
The key ablation isolates seed content: the identical generator and channel with raw versus primitive versus concatenated seeds. A second ablation replaces $p$ with a random projection of the raw feature to the same dimension. This rules out the explanation that low dimensionality alone, rather than clinical meaning, produces robustness.
Why not just…?
Why not just write the predicted findings into the prompt as text? This is the most serious threat, and it is built into the design as the serialization channel. The case against assuming it suffices is TheoremQA's finding that supplying rules in text did not help models apply them. Radiology findings are simpler than theorems, though, and text serialization may well match state seeding. If it does, we will say so. The surviving claim is then the content claim and the site-probe diagnosis, and practitioners get the simpler recipe.
Why not just train on data from several hospitals? QIAI showed that mixed-institution training restores the multimodal advantage. That requires labeled data from the deployment site, which is often unavailable when a model moves to a new hospital. We report mix-split as an oracle. Our question is how much of its benefit a site-invariant seed recovers with zero target-site data.
Why not just use a large vision-language model that reads the image directly? LuminaECG reports that grounded primitives beat generalist scale for ECGs. On PTB-XL, GPT-4o and Claude 3.5 Opus recovered under 10.3% of critical findings against 50.8% for the grounded 2B model. That evidence comes from a different modality and does not settle radiology. The fine-tuned patch-token baseline tests this directly. If it matches the primitive seed out of distribution, the bottleneck is unnecessary, and our probe analysis should show that its representations already discard site identity.
Why not just retrieve knowledge-graph facts instead of extracting primitives? KMQA's error analysis attributes most residual failures to missing or noisily retrieved graph content. Retrieval also says nothing about the specific image in front of the model. A CNet-NIC-style arm, with concept embeddings retrieved from predicted findings, is included as a variant of the concatenated seed.
Proposed Experiments and Decisive Checks
All models train on MIMIC-CXR v2.0 (91,544 training reports) and are evaluated on the in-distribution MIMIC validation set, on Indiana University Chest X-Rays, and on the Stanford test reports. Stanford is text-only, so it tests whether visual seeding harms the model when no image is present.
Metrics are BLEU and ROUGE-2 for comparability with QIAI. Because these reward wording more than correctness, we add finding-level agreement between generated and reference impressions. This is scored with a report labeler independent of the one used to train the primitive extractor, which avoids circular evaluation. A radiologist also reviews a blinded subset of 300 Indiana studies.
The decisive check has four parts. Primitive seeding must beat text-only on Indiana while raw seeding falls below it. The out-of-distribution drop must correlate positively with site-probe accuracy across the grid, including the random-projection control. The extractor's calibration error on Indiana must be measured, so that failures can be attributed either to the generator trusting the seed or to the seed itself being wrong.
As a secondary replication outside radiology, we apply the same content ablation to ECG finding generation. Seeds are built from delineated P, QRS and T measurements, training uses MIMIC-IV-ECG, and testing uses PTB-XL and Chapman-Shaoxing. These are the cohorts LuminaECG used, and its reported 50.8% PTB-XL critical-finding recovery is the reference point. If the site-probe relationship holds in both modalities, the result is a general rule for choosing what to put in a generator's initial state when the deployment distribution differs from training.
References
- QIAI at MEDIQA 2021: Multimodal Radiology Report Summarization
- Diagnosing as Cardiologists Do: ECG Agents with Doctor-Grounded Priors for Clinical Reasoning Across Diseases and Populations
- Incorporating Global Visual Features into Attention-based Neural Machine Translation
- Multi-modal gated recurrent units for image description
- Multimodal Grounding for Sequence-to-Sequence Speech Recognition
- Improving Image Captioning by Leveraging Knowledge Graphs
- Towards Medical Machine Reading Comprehension with Structural Knowledge and Plain Text
- TheoremQA: A Theorem-driven Question Answering Dataset