Letting Report Structure Decide What Survives Downsampling

Chest X-ray report generators read a high-resolution image, shrink it into a coarse grid of features, and then write a report from that grid. Small findings such as a thin pneumothorax line, a nodule a few millimetres wide, or a misplaced tube can be averaged away during that shrinking, long before the text decoder looks at the image. The problem gets worse when the model is moved to a new hospital. Unfamiliar scanners and image processing make the visual features less reliable, and the decoder falls back on its language habits: it writes fluent, generic "lungs are clear" reports that are clinically wrong. We propose to let the structure of the report steer the image encoder. A radiology report has a predictable layout: a Findings section with sentences about the heart, lungs, pleura, bones and devices, followed by an Impression. Each of these topic slots becomes a query that tells the encoder's downsampling layers which pixels to keep for that slot. The result is a separate, slot-specific summary of the image, built before the fine detail is lost.

The core claim, stated for a colleague: conditioning the downsampling operations inside the vision encoder on the report's section-and-topic schema keeps small, report-relevant evidence that late cross-attention cannot recover. We expect this to matter most under cross-hospital shift, because anatomy and report organization stay stable across sites while pixel statistics do not. We test this in the established train-on-MIMIC-CXR, test-on-IU-X-Ray protocol. The comparisons are against strong flat, hierarchical and domain-generalized baselines, and, most importantly, against a compute-matched high-resolution model that uses ordinary cross-attention.

What exists and what is missing

The closest work splits along the four directions without joining them where it matters.

RIHA builds a visual feature pyramid from ResNet-101 stages and aligns it with paragraph-, sentence- and word-level text features through an optimal-transport loss. It supports our premise that flat reports hurt alignment. However, its pyramid comes from standard, text-blind strided downsampling, its alignment module is dropped at inference, and it was never tested across hospitals.

ARDGen is the domain-generalization reference point. It trains on MIMIC-CXR and tests on IU X-Ray using consistency regularization and an auxiliary decoder, but it decodes a flat report from a standard downsampled grid.

Two methods add structure on the text side. ORGAN routes decoding through a pruned observation tree. This constrains our design: hard pruning means a lesion missed upstream can never be described, so we use soft structure only. PromptMRG prepends flat diagnosis tokens. Both leave the image encoder untouched.

On the visual side, AGFNet preserves local detail with 14 detector-defined anatomical boxes. That approach misses findings outside the boxes and depends on a detector that is itself vulnerable to site shift. The 3D systems Argus and BTB3D preserve resolution through token folding and wavelets, but then feed a flat language model and do no cross-site adaptation.

Three papers supply our mechanisms. LIP replaces fixed pooling with a learned importance map and improved small-object detection on COCO (Faster R-CNN small-object AP from 21.9% to 24.0%). LIP's maps are driven only by the image, though; nothing tells them what the downstream text needs. Gated cross attention from Kim & Shin gives us a learned mixing gate whose value we can read off as a measure of how much the decoder relies on the image. HIBRIDS and TreeJC supply the text-structure tools: soft tree-distance attention biases and ancestor-chain conditioning. Buchmann et al. motivates making structure explicit, since implicitly learned structure was less precise. The size of the deployment problem is shown by AMOS, which reports drops of over 20% on unseen MRI scanners.

Core Claim

Each report slot $k$ (for example "Findings › Pleura") has an embedding $s_k$. Following TreeJC, $s_k$ is formed from its ancestor chain of section and topic. At each downsampling stage of the encoder, we replace strided convolution with LIP-style importance pooling whose logit network is modulated by $s_k$ through a per-channel scale and shift:

$$ z_k(p) \;=\; \sum_{q \in \Omega(p)} \frac{\exp\big(F_\theta(X;\, s_k)_q\big)}{\sum_{r \in \Omega(p)} \exp\big(F_\theta(X;\, s_k)_r\big)}\, X_q . $$

In this equation, $X$ is the feature map entering the stage, $\Omega(p)$ is the pooling window that maps to output location $p$, $F_\theta$ is the learned importance-logit network, and $z_k$ is the slot-specific downsampled map.

Early full-resolution stages are shared across slots. Only the later, already-reduced stages run once per slot, batched along the batch dimension. The decoder writes the report slot by slot. Tokens in slot $k$ cross-attend mainly to $z_k$ through a gated cross-attention layer. Within the text, HIBRIDS-style learnable biases indexed by slot-tree distance let sibling slots share context softly, with no pruning.

The schema is fixed (about six Findings topics plus Impression), so it is always available at test time and nothing has to be parsed from the input. During training, MIMIC-CXR Findings sentences are assigned to topics by a keyword rule we will write and audit on a hand-checked sample. Sentences that cannot be assigned go to an "other" slot.

The claim is falsifiable in a specific way. If the gains come from conditioned early downsampling, they should be concentrated on small-extent findings, be larger on the external hospital than in-distribution, grow as input resolution is artificially degraded, and vanish when the conditioning is removed while everything else stays fixed.

Research Program

The work proceeds in three stages.

The first stage builds the slot schema and the topic assignment for MIMIC-CXR and IU X-Ray. Assignment for IU is needed only to score per-slot outputs, never for inference.

The second stage implements text-conditioned importance pooling in a ResNet-101 backbone, so the visual capacity matches the RIHA, AGFNet and ARDGen comparisons. We will run at a higher input resolution than the usual 224 pixels, chosen so total FLOPs can be matched by baselines.

The third stage analyzes behaviour under shift, not only scores. We will record three diagnostics. The first is the mean gate value per slot on MIMIC versus IU, which shows whether the decoder abandons the image out of distribution. The second is image sensitivity, measured by swapping the input image for one from another patient and computing how much the generated report changes. The third is whether the learned importance maps land on the annotated finding region. Since the file names no CXR localization dataset, this last check uses a small hand-annotated subset that we create.

Baselines

The naive reference point is a ResNet-101 grid encoder with a flat Transformer decoder. It is trained on MIMIC-CXR and evaluated zero-shot on IU X-Ray, with no structure, gating or regularization. Its main job is to quantify the cross-site gap and the rate of generic normal reports.

PromptMRG is the strongest flat method with a number in the file. It reports clinical-efficacy F1 of 0.211 on IU X-Ray (PromptMRG).

ARDGen is the strongest domain-generalization competitor. It reports an "11.8% improvement in F1-score and a 7.5% boost in Precision over previous top-tier models like PromptMRG" on IU X-Ray as the unseen domain. The file gives no absolute ARDGen F1, so no reported result was found on Lacuna for ARDGen's absolute IU X-Ray F1. We will rerun it under our protocol. Beating ARDGen means a statistically reliable gain in clinical F1 on IU X-Ray over our rerun, with BLEU and ROUGE-L not lower, using bootstrap confidence intervals over reports.

RIHA is the strongest hierarchical competitor. It has only been reported in-distribution, for example MIMIC-CXR clinical F1 of 0.375. We will run it in the cross-site protocol to test whether text hierarchy alone helps under shift.

All methods share the same pretrained backbone initialization, training data, input resolution where applicable, training steps and a fixed hyperparameter search budget. Model selection uses only the MIMIC-CXR validation split, never IU X-Ray.

Three ablations isolate the key ingredient. Unconditioned LIP pooling keeps the same pooling layers but removes the slot query, which tests whether the text conditioning matters at all. Late-only slot queries keep the slot structure but cross-attend to an ordinary grid, which tests whether conditioning must happen before downsampling. A compute-matched high-resolution flat model is the most dangerous baseline and is discussed below. A further variant is the full model with ARDGen's consistency regularization added, to check whether the two are complementary.

Why not just…?

Why not just feed a higher-resolution image and let ordinary cross-attention pick the details? This is the real threat. For chest X-rays, a 32×32 final grid is cheap to cross-attend, and the 3D results in Argus show that keeping more tokens can beat larger models. Our counter-argument is that cross-attention operates on a grid whose cells were already formed by text-blind pooling. LIP's small-object gains suggest that what is discarded in early pooling matters. That is only an argument, so we settle it empirically. The flat model gets the higher resolution and backbone width needed to equal our total FLOPs, and the late-only slot-query ablation separates "structured queries" from "conditioned downsampling." If the compute-matched flat model ties us on IU X-Ray, especially on small findings, we will report that the claim fails.

Why not just use a large pretrained vision-language model with heavy augmentation? ARDGen reports that conventional MIMIC-trained models, PromptMRG among them, hallucinate institution-specific phrasing such as "compared to previous study" on IU X-Ray. Scale does not obviously fix that. Augmentation addresses pixel statistics but not the decoder's tendency to ignore a degraded image. We include ARDGen as the representative of regularization-based robustness and check whether our method adds to it rather than replaces it. We do not claim to beat frontier general-purpose models. Our question is whether conditioning downsampling on report structure helps at a fixed model size.

Why not just prepend flat diagnosis prompts or use detector boxes? PromptMRG's prompts depend on an image classifier that runs on the same downsampled grid, so a missed small finding is missed in the prompt too. ORGAN's tree pruning makes this failure irreversible. AGFNet's 14 detector boxes cannot represent findings outside them, and the detector is itself exposed to site shift. Per-finding F1 broken out by lesion extent will show whether these methods lose specifically on small and off-box findings, as we predict.

Proposed Experiments

The main experiment trains every model on MIMIC-CXR and evaluates it on the MIMIC-CXR test split and on IU X-Ray zero-shot. We report CheXpert-label clinical-efficacy precision, recall and F1, as used throughout this literature, together with BLEU-4, METEOR and ROUGE-L. On top of these metrics we add the generic-normal-report rate, per-slot gate values and image-swap sensitivity.

The first decisive check is a breakdown by finding size. We predict our gains over the compute-matched flat model concentrate on small-extent labels (pneumothorax, nodule or lesion, fracture, support devices) and are near zero for large findings such as cardiomegaly.

The second decisive check is controlled resolution degradation. IU X-Ray inputs are downsampled and re-upsampled at several factors to mimic lower-quality acquisition. Our method's lead should grow with degradation if it truly preserves small evidence, and it should not if the gain is just regularization.

The third check is the gate diagnostic. On IU X-Ray, the flat gated baseline's image gate should drop relative to MIMIC, while ours should drop less, and image-swap sensitivity should follow the same pattern.

Together, these checks separate "structure helps under shift" from "more compute helps." They also give a clear negative result if conditioning on report structure turns out not to matter.