Does the Search Objective Change Which Fusion Architecture Wins? Clinically Scored Architecture Search for Chest X-ray Reports

Systems that write chest X-ray reports must combine several inputs: frontal and lateral views of the current exam, an earlier exam of the same patient, and the clinical question that prompted the scan. Where and how these inputs get mixed inside the network is usually chosen by hand. Neural architecture search (NAS), which means automatically trying many network wirings and keeping the best, can make this choice instead. So far it has always ranked candidate wirings by generic scores such as validation loss, accuracy, or recall. Radiology evaluation research says those scores are the wrong yardstick. A report can be fluent and score well on word overlap while inventing a finding or putting it on the wrong side of the chest. Metrics that parse reports into graphs of clinical entities and relations track radiologists' judgments far better.

This note proposes ranking fusion wirings by how clinically correct their reports are, using entity-and-relation graph overlap and a composite error metric, instead of by training loss. The first question is empirical and comes before any method claim: do the two yardsticks actually prefer different architectures? If they agree, this line of work is unnecessary, and we would want to know that cheaply. If they disagree, there is a second, less obvious argument for clinically scored search. Choosing among a few hundred discrete wirings carries far less freedom than tuning billions of weights against a metric. Architecture-level optimization should therefore be much harder to "game" than the preference fine-tuning that currently uses these metrics. The proposal tests that argument directly by checking the selected wirings against clinical metrics the search never saw.

Core Claim

We claim three things. First, on chest X-ray report generation with multi-view and prior-exam inputs, the ranking of candidate fusion wirings under cross-entropy loss and under a clinical graph metric differs enough to change which architecture is selected. Second, wirings selected by the clinical metric improve clinical correctness on metrics that were held out from the search, not only on the metric they were chosen by. Third, these gains add to what weight-level preference tuning provides rather than duplicating it.

The closest work does one half or the other. Multimodal NAS on medical data includes MUFASA, an evolutionary search over per-modality branches and a fusion module for health records, and the ADReSS dementia NAS study, which searched an audio branch and picked the fusion operator by hand. Both rank candidates by classification scores. Metric-guided training includes CheXalign, which turns automated clinical judges into preference pairs for fine-tuning a fixed vision-language model, and HiMed-RL, which uses a token/concept/semantic reward hierarchy for reinforcement learning. Both change weights and never wiring. Report generators with hand-designed multi-view and prior-exam fusion, MLRG (cross-attention over views and prior studies) and CoCa-CXR (spatially masked regional cross-attention between current and prior images), are evaluated with clinical metrics but not searched with them.

Research Program

Search space. The inputs are the current frontal and lateral images, the most recent prior image when one exists, and the indication text. These follow MLRG's input set, including its learned "no prior indication" token for missing text. Each input has its own encoder lane, as in MUFASA's separate branches. Two decisions are searched. The first is where each pair of lanes exchanges information, chosen from a set of encoder depths, with the asymmetric skip paths that MFAS found useful allowed. The second is how they exchange it, chosen from the primitive operators used by BM-NAS: cross-attention, gated linear units, addition, concatenation, and low-rank bilinear pooling. To give CoCa-CXR's design a fair place in the space, a regional masked cross-attention operator for the current-versus-prior pair is also included.

The image backbone is pretrained, and the decoder follows the R2GenCMN and ViLMedic codebases. Keeping these fixed keeps the comparison about fusion only.

Search strategy. Report-level metrics cannot be differentiated, so we use tournament-selection evolution in the style of MUFASA. Candidates inherit weights from a shared supernet, as in the knowledge-sharing NAS method. Each candidate is fine-tuned briefly and then judged by decoding reports on a fixed validation subset of MIMIC-CXR.

The fitness of candidate architecture $a$ is

$$F(a) = \mathrm{RG}(a) - \lambda \, \max(0, L(a) - L_{\mathrm{ref}}),$$

where $\mathrm{RG}(a)$ is the RadGraph F1 of its decoded validation reports. RadGraph F1 is the overlap between the reference and generated graphs of anatomy and observation entities and their relations, extracted with the RadGraph parser. $L(a)$ is mean report length, $L_{\mathrm{ref}}$ is mean reference length, and $\lambda$ is a penalty weight. The length term is there because CheXalign documented verbosity gaming of clinical judges. A second run uses RadCliQ from Evaluating Progress in Automatic Chest X-Ray Radiology Report Generation as the fitness. RadCliQ combines BLEU and RadGraph to predict how many errors radiologists would find.

Analysis. Every finalist is scored with the RadCliQ error taxonomy (false findings, omissions, wrong location, wrong severity). This lets us ask which wiring choices cause which kinds of error. One example hypothesis: fusing the prior exam early should reduce errors about change over time, which we can probe on CoCa-CXR's MS-CXR-T progression benchmark.

Baselines

The naive reference points are a plain encoder-decoder transformer run in the R2GenCMN codebase and the random-retrieval baseline. The retrieval baseline matters because Evaluating Progress in Automatic Chest X-Ray Radiology Report Generation reports that leading generators beat random retrieval by only 5.5% under RadCliQ. Any searched model must clear that low bar by a clear margin.

The strongest existing hand-designed competitors are MLRG and CoCa-CXR, because they use the same multi-view and prior-exam inputs. MLRG reports only relative gains, such as a 2.7% RadGraph F1 improvement on two-view studies, so we have no absolute MLRG figure to cite. CoCa-CXR reports RadGraph F1 of 24.2% on MIMIC-CXR report generation, and this is the number to beat. Both models will be re-implemented inside our encoder and decoder, with the same backbone, training data, and total GPU budget as one search run plus final retraining. We will also report each system's own published configuration. Beating them means a statistically significant gain, judged by bootstrap over test studies, on held-out clinical metrics and not only on RadGraph F1.

The ablation that isolates the key ingredient is the identical search with cross-entropy validation loss as fitness. It uses the same space, seeds, and budget, so the only difference is the yardstick. Random search with clinical fitness separates the value of the yardstick from the value of evolution.

For post-training, CheXalign-style preference tuning (Direct Preference Optimization with a length-controlled clinical judge) is applied to the best hand-designed model. CheXalign reports a 14% CheXbert improvement on CheXagent-8B, but on a different backbone, so we compare only within our setup. That tuning is then applied to the searched model to test whether the gains add up.

Why not just…?

Why not just keep a hand-designed architecture and preference-tune it with the clinical metric? This is the most serious alternative, because CheXalign already gets large clinical gains this way. Its own need for length control and HiMed-RL's need for multi-level verifiers show that weight-level optimization against these metrics tends to drift toward gaming them. Weight tuning also cannot recover information that a fixed fusion bottleneck throws away, a concern raised by LLM-CXR. The four-way comparison settles this: hand-designed versus searched, each with and without tuning. If tuning the hand-designed model matches searched-plus-tuned on held-out metrics, then the search is not worth its cost.

Why not just train a handful of hand-designed fusion variants and pick the best by RadGraph F1? This could capture most of the benefit, and we treat it as a real threat. We will run it explicitly: about ten designs taken from MLRG, CoCa-CXR, early fusion, and late fusion, each selected by clinical fitness. If evolution over the full space does not beat this curated pick, we will report that the useful contribution is clinically scored selection, not search, and recommend the cheap version.

Why not just fine-tune a large generalist vision-language model? The MIMIC-RRS study found that general-purpose BART-Large beat its biomedical counterpart, and Fusion or Confusion? questions whether added multimodal complexity helps at all. On the other side, HiMed-RL reports a 3B model beating MedGemma-27B and InternVL3-14B on clinical metrics. We will include a fine-tuned generalist under equal training compute as a ceiling check. This also makes the proposal's answer to the generalist question explicit rather than assumed.

Why not just use CheXbert disease labels as fitness, since they are cheaper and less noisy? The low radiologist agreement on RadGraph relations (Cohen's Kappa 0.397, per RadGraph) makes this a fair challenge. Disease-label fitness becomes a third fitness variant. Presence and absence labels cannot see location or severity errors, so we predict it will underperform on those error types specifically, and the per-error-type analysis will show whether it does.

Proposed Experiments

The first and decisive check is a ranking study. We train about one hundred randomly drawn wirings to convergence on MIMIC-CXR and compute Kendall tau between their rankings under validation loss and under RadGraph F1. If tau is very high and the top-five sets mostly coincide, we stop. At that point the claim is falsified, and that negative result is useful to the multimodal NAS community.

If the rankings diverge, we run the full searches and evaluate on the MIMIC-CXR test split with metrics never used in fitness: GREEN, CheXbert F1, and RadCliQ for the RadGraph-guided run, or RadGraph for the RadCliQ-guided run. The gap between the fitness metric and the held-out metrics measures how much the search overfit its own yardstick. We compare that gap against the same gap for preference tuning, which tests the "architecture is harder to game" argument directly.

Generalization is tested by transferring the selected wirings, retrained without any new search, to IU X-Ray and the out-of-domain Padchest-GR benchmark. HiMed-RL reports Padchest-GR gains only relative to the second-best baseline, so we will establish absolute reference numbers ourselves. Temporal fusion is probed on MS-CXR-T, where CoCa-CXR reports 65.0% average accuracy. Finally, a small reader study with the RadCliQ error taxonomy on 100 test studies checks whether the metric-level gains correspond to fewer radiologist-identified errors, which is the outcome all of this is meant to serve.