Finding Trees as a Language-Neutral Bridge for Zero-Shot Multilingual Radiology Reports

Most hospitals that would benefit from automatic chest X-ray report drafting do not write in English, yet nearly all paired training data, including MIMIC-CXR, is English. This proposal asks one question: can a model trained only on English reports write faithful reports in Spanish, or in another target language, if it first decides what the image shows in a format that belongs to no language? That format is a "finding tree", a small structured plan listing anatomical regions, the findings in each, and their attributes (present or absent, left or right, severity, change since the prior study). A separate multilingual text model then writes that plan out as prose in the requested language. This is the plan-then-write idea from story generation, applied to cross-lingual transfer.

The same tree also repairs a known flaw in how such image models are pretrained. Contrastive learning, which trains an image encoder by pulling matched image–text pairs together and pushing every other pair in the batch apart, wrongly pushes apart two different patients who have the same findings. This is the "false negative" or "semantic collision" problem, and it is endemic in radiology because so many studies are normal. Finding trees are language-neutral codes, so two studies with the same tree can be treated as positives whether their reports are in English or Spanish. Neither the existing collision fixes nor the existing cross-lingual planners provide this.

Core Claim

A single hierarchical, language-neutral content plan can serve two roles: it defines which pairs count as negatives during contrastive pretraining, and it is the only interface between image understanding and target-language writing. We claim this combination yields target-language reports whose clinical content matches English-language performance much more closely than end-to-end multilingual generation does. It should also make fewer negation and laterality errors than generating in English and then machine-translating. The hierarchy matters. A flat list of entities cannot express "no effusion on the left", and negation and side are the errors that hurt patients.

The closest prior system is μPLAN. It predicts an ordered sequence of Wikidata entities and then writes a target-language summary, and it showed that plans help zero-shot cross-lingual summarization. Its plans are flat, it works on text-to-text Wikipedia and news, and its plans play no part in representation learning. Our plans are trees with clinical attributes, are predicted from images, and also drive the contrastive objective. On the medical side, HTSC-CIF decomposes report generation into hierarchical levels with an entity contrastive loss, and CMCA subtracts features of retrieved similar cases to suppress dominant normal phrasing. Both decode English directly, with no language-neutral intermediate, so neither can transfer to another language.

Research Program

Finding trees. A tree has three levels: anatomical region, finding, and attribute slots for presence, laterality, severity, and temporal change. Trees are extracted from English MIMIC-CXR reports only, using the existing English labeling tools behind MIMIC-CXR. That paper reports CheXpert-labeler mention F1 of 0.973 for pleural effusion and 0.956 for pneumonia, so the extractor is reliable for common findings. No target-language parser is ever needed, which matters because Gap 1 in our file is precisely that ontology tools such as MetaMap and RadGraph exist only for English. We will audit extraction noise on hedged and negated statements, a limitation the file flags for these parsers. We will report how often the trees misencode negation, because those errors propagate into both the contrastive targets and the plans.

Collision-aware pretraining. Take a batch of $N$ studies. Let $s_{ij}\in[0,1]$ be the weighted overlap between the finding trees of studies $i$ and $j$, with attribute mismatches such as opposite laterality counted as disagreement. The image encoder is trained against soft targets

$$q_{ij} = \frac{\exp(s_{ij}/\tau)}{\sum_{k=1}^{N}\exp(s_{ik}/\tau)}, \qquad \mathcal{L} = -\sum_{i,j} q_{ij}\,\log p_{ij},$$

where $p_{ij}$ is the model's softmax over image–plan embedding similarities and $\tau$ is a temperature. Two normal studies therefore become near-positives instead of forced negatives. The loss reuses the soft-target idea of MedCLIP, which built similarities from UMLS concept overlap, and of Anatomical Structure-Guided Pre-training, which used structured triplets. Our variant differs in two ways. The similarity is computed on trees that encode attributes, rather than on bags of concepts. And the second side of each pair is a plan embedding, not English text, so the learned space contains no language at all.

Staged adaptation. Following the staging evidence from Don't Stop Pretraining and TALLRec, training proceeds in four stages:

  1. collision-aware image–plan pretraining on MIMIC-CXR;
  2. unlabeled image-only adaptation on target-hospital radiographs from PadChest, using no target text, to absorb scanner and population shift;
  3. training the planner, a decoder that emits the tree from image features;
  4. training the realizer, which writes the tree as prose.

The realizer starts from a multilingual instruction-tuned model, mT0 from Crosslingual Generalization through Multitask Finetuning. It is adapted with low-rank adapters on English plan-to-report pairs only, so that its multilingual ability is not overwritten. The staging is a design constraint, not a headline claim. We test it against pooled training because the file shows ordering effects are real and also brittle.

Length control. The generative trade-off that paper reports, where multitask instruction tuning makes long outputs too brief, is a direct threat here. Radiology reports must state negatives as well as positives. We use the tree as a coverage contract: decoding is constrained so that every plan node is realized, and node coverage is measured directly. This tests the file's Tension 1 conjecture that an explicit plan limits truncation independently of how the realizer writes.

Baselines

The naive reference point is a multilingual vision-language model fine-tuned end-to-end on English MIMIC-CXR image-report pairs, then prompted to write in Spanish. It uses the same image encoder initialization, the same realizer backbone, and the same number of training tokens and hyperparameter search trials as our system. Analyzing and Mitigating Cross-Lingual Degradation in Multilingual Medical VQA motivates expecting this baseline to degrade: it documents systematic non-English drops on Qwen3.5, Gemma3, and InternVL3.5, worst on attribute-aware reasoning.

The strongest competitor is the cascade: our own best English report generator, with the same planner and realizer but English output, followed by a strong general machine-translation system into Spanish. It is the strongest because radiology prose is formulaic and translation quality is high. It also shares every component of our system except where the language switch happens, which makes the comparison fair by construction.

We also include a published English generator, CMCA, run through the same translation step. CMCA reports a CheXpert F1 of 0.356 on MIMIC-CXR against a previous best of 0.303 (CMCA). For IU-Xray it reports BLEU-4 of 0.215 and ROUGE-L of 0.392, while HTSC-CIF reports BLEU-4 of 0.192 and METEOR of 0.253 (HTSC-CIF). In English, our pipeline should at least reach CMCA's 0.356 CheXpert F1 on MIMIC-CXR before any cross-lingual claim is meaningful. There is no reported result found on Lacuna for any generation model on PadChest-GR, so all target-language numbers will be new.

On the representation side, the reference is MedCLIP's soft-target loss over UMLS concept bags, retrained on our data with the same encoder. It reports 0.594 zero-shot accuracy on CheXpert-5x200, against 0.418 for ConVIRT and 0.432 for GLoRIA (MedCLIP). Standard InfoNCE without debiasing is included as the naive floor.

Two ablations isolate the key ingredients:

  • Flat plan. Replace the tree with a μPLAN-style flat ordered list of finding entities, which drops the attribute slots and hierarchy. This tests whether hierarchy, rather than planning as such, carries the gain.
  • Contrastive-only change. Keep the tree planner and realizer but pretrain with plain InfoNCE. This tests whether collision-aware pretraining matters once a planner exists.

Beating the cascade means a statistically significant gain in target-language clinical F1 on PadChest-GR, together with a lower rate of negation and laterality errors, at no loss in fluency judged by native-speaker radiologists. A gain in BLEU alone does not count.

Why not just…?

Why not just generate in English and machine-translate? This is a real threat, and we do not assume it loses. The cited evidence is indirect. μPLAN found cascades propagate errors in summarization, and Zero-shot Reading Comprehension found translation pipelines underperform latent transfer. Both are general-domain results. The specific failure we predict is that translation flips or drops negations and sides in formulaic sentences. PadChest-GR's 3,422 manually annotated negative finding sentences and its box-linked positive sentences let us count exactly these errors. If the cascade ties our system on both clinical F1 and negation and laterality error, the honest conclusion is that planning helps only in English. In that case the contribution shrinks to the representation result, and we will report it that way.

Why not just fine-tune a multilingual vision-language model end-to-end and prompt in Spanish? The multilingual medical VQA study shows these models lose accuracy outside English, most on attribute-level reasoning. Its training-free fix, steering vectors, needs paired English–target activations, which violates the zero-shot setting. The end-to-end baseline above measures the drop directly on report generation. Comparing it with our system at matched compute separates "multilingual backbone" from "language-neutral bottleneck".

Why not just use MedCLIP's concept-overlap soft targets, or a flat entity plan? MedCLIP's similarity is a concept bag, so "left effusion" and "no right effusion" can share concepts. A flat plan loses the same distinctions at generation time. Both are run as ablations under identical budgets. If neither tree similarity nor hierarchy improves attribute-level errors, the tree is unnecessary decoration, and the simpler variant wins.

Why not pool all data in one stage? The ordering evidence from Neural Machine Translation Training in a Multi-Domain Scenario and DAPT/TAPT favors staging, but it comes from text-only settings. We run the pooled variant at equal total steps. If it matches the staged pipeline, we drop staging.

Proposed Experiments

Training and evaluation data. Training uses English MIMIC-CXR reports and images, plus unlabeled PadChest images for stage two. The primary zero-shot test set is PadChest-GR (PadChest-GR). It has 4,555 studies with paired Spanish and English reports, positive finding sentences linked to bounding boxes, explicit negative sentences, and progression tags.

Metrics. The primary metric is plan-level clinical F1 against trees parsed from PadChest-GR's English reference reports. This is language-free and measures what the model decided. It is reported alongside target-language clinical F1, computed by mapping generated Spanish sentences to finding labels. That mapping is done two ways, back-translation followed by the English labeler and a small radiologist-annotated sample, because each check has a different confound. Further metrics are:

  • negation-flip and laterality-flip rates on the annotated negative sentences;
  • grounding accuracy of each plan node's attention against PadChest-GR's boxes, which measures the dense-collision issue at the patch level;
  • node coverage and report length relative to the Spanish references;
  • ROUGE-L and BLEU against the Spanish references.

English sanity checks. On MIMIC-CXR and IU-Xray we compare against CMCA's and HTSC-CIF's reported numbers. On CheXpert-5x200 and RSNA Pneumonia we test whether tree-based soft targets match or exceed MedCLIP's 0.594 zero-shot accuracy and BioViL's 0.831 zero-shot AUROC (BioViL).

Second language. To check generality beyond Spanish without target reports, we test the image-side representation in additional languages on the multilingual medical VQA benchmark. There we compare plan-conditioned answering against MedVL-XLRepE's reported gains of up to 6.33%, while noting that it uses paired calibration data and we do not.

Decisive checks. Three results would settle the proposal. First, collision-aware tree targets should raise planner F1 most on normal and near-normal studies, where collisions concentrate. If gains are uniform, the collision story is wrong. Second, the gap between English plan F1 on MIMIC-CXR and on PadChest-GR should be explained by image shift, which stage two should shrink, and not by language. By construction, language cannot enter the plan. Third, against the cascade, our system must reduce negation and laterality flips in Spanish output. That outcome is the one that justifies putting the language switch after the plan rather than after the English text.