Capacity-Limited Word-to-Patch Transport for Label-Efficient Chest X-ray Pretraining

Medical image models trained on paired X-rays and radiology reports learn which words go with which parts of the image. That matching is what makes them useful for segmentation and detection when only 1% of images carry expert labels. Today the matching step is local and greedy. Each word independently picks the image patches most similar to it, so a few generic patches, such as the large, uniform lung field, attract almost every word. Many words then collapse onto the same region. The cross-lingual and knowledge-graph alignment literature calls this "hubness," and it answers the problem with global assignment: solve for the best joint matching of all words to all patches at once, under a limit on how much each target can absorb.

We propose to replace the word-to-patch attention inside medical vision-language pretraining with a partial, capacity-limited optimal transport problem. Optimal transport here means finding the cheapest way to move a fixed amount of "matching mass" from words to patches. The two constraints that matter are these. Each patch can receive only a bounded share of mass, which is what defeats hubs. Words may leave part of their mass unmatched, which lets report phrases with no visual counterpart ("compared to prior," "no pneumothorax") stay unassigned instead of being forced onto background tissue. The testable claim is specific. The patch-capacity constraint, not sharper or sparser attention, is what improves dense low-label transfer. It should show up as measurably lower hubness in the learned word-patch maps.

Core Claim and Positioning

The closest work splits into three groups, and none places a capacity constraint at the word-to-patch level during pretraining.

The first group supplies the pretraining setup we modify. GLoRIA pulls each word toward a softmax-weighted average of image sub-regions. Because the softmax is normalized separately for each word, nothing stops every word from attending to the same region. MGCA keeps unconstrained bidirectional cross-attention at the token level. It uses Sinkhorn normalization only to assign whole images and reports to a small set of latent disease prototypes. Sinkhorn is the iterative row-and-column rescaling that turns a similarity matrix into a transport plan. MGCA therefore has global assignment at the coarsest granularity and none at the finest.

The second group uses optimal transport but in a different place. OTCXR applies balanced transport between dense feature maps of two augmented views of the same image, so it never touches text, and its authors report sensitivity to the marginal constraints. OTDRG transports patches onto 14 fixed disease anchors for report generation. That is a closed vocabulary rather than the report's own words.

The third group supplies the key constraint and a warning. MADPOT shows that balanced transport forces healthy background patches onto anomaly prompts, and that partial transport fixes this. It does so only as a downstream adapter on frozen CLIP. It also reports that partial transport needs contrastive regularization to avoid collapse, which our loss design respects.

On the assignment side, several papers motivate the design:

We treat report phrases with no visual referent exactly like these "dangling" entities.

The Objective

For one image-report pair, let $w_1,\dots,w_n$ be word embeddings and $v_1,\dots,v_k$ patch embeddings. Let $C_{ij} = 1 - \cos(w_i, v_j)$ be the matching cost, $a \in \mathbb{R}^n$ the word masses (uniform over content words), $b \in \mathbb{R}^k$ the patch capacities (uniform, scaled by a slack factor $\rho \ge 1$), $m \in (0,1]$ the total mass to be moved, and $\epsilon$ an entropy weight. We solve

$$P^\star = \arg\min_{P \ge 0} \langle P, C\rangle - \epsilon H(P) \quad \text{s.t.} \quad P\mathbf{1} \le a,\; P^\top \mathbf{1} \le \rho\, b,\; \mathbf{1}^\top P \mathbf{1} = m .$$

We solve it with Sinkhorn iterations after adding one "unmatched" slot on each side, the standard reduction of partial transport to balanced transport. The local similarity of an image-report pair is $s = -\langle P^\star, C\rangle / m$. This score enters an InfoNCE loss over the batch, which contrasts each image's score with its own report against scores with other reports. That loss replaces GLoRIA's attention-based local term. MGCA's global and prototype terms are kept unchanged, so the only thing that changes is the local matching operator.

The parameter $\rho$ sets how much a single patch may absorb. At $\rho \to \infty$ the patch constraint vanishes and the method reduces to word-wise attention with a discard option. That limit is the central ablation.

Research Program

We start from MGCA's setup: a ViT-B/16 image encoder, a BioClinicalBERT text encoder, and pretraining on MIMIC-CXR as described by MIMIC-CXR. Only the local term changes.

Three design questions get direct experiments.

  1. How to set the total mass $m$. We compare a fixed fraction swept over a grid with a per-report rule. The rule sets $m$ in proportion to the fraction of non-negated finding tokens, so "no acute cardiopulmonary process" moves little mass.
  2. How strict patch capacity should be. We sweep $\rho$ from 1 to effectively infinite.
  3. How to differentiate. We compare backpropagating through about 20 unrolled iterations with treating $P^\star$ as a constant. Treating it as constant is valid for the transport cost by the envelope theorem.

The computational load is modest. Each pair is a problem of roughly 100 words by 196 patches, and the batch needs $B^2$ of them, the same scaling GLoRIA already pays. The iterations are dense matrix products that run in parallel on a GPU, as UPL-EA notes for its own Sinkhorn step.

Alongside the method we build a diagnostic. For each pretrained model, we compute the "k-occurrence" skew of patches over MIMIC-CXR reports: how unevenly the patches appear among the top-k matches of words. This is the standard hubness statistic from lexicon induction. We then ask whether it predicts 1% segmentation Dice across all models we train. If hubness does not track dense transfer, the motivation for this project is wrong, and we would report that.

Baselines

The do-nothing reference is an ImageNet-initialized encoder fine-tuned with the same 1% labels. GLoRIA reports 74.4 AUROC for it on CheXpert at 1%. It shows how much any pretraining buys.

The local-attention reference is GLoRIA itself. It reports 86.6 AUROC on CheXpert at 1% and 0.358 Dice on SIIM at 1%. It is the purest case of the unconstrained matching we replace.

The strongest directly comparable method is MGCA. It reports 88.8% AUROC on CheXpert at 1%, 49.7% Dice on SIIM pneumothorax at 1%, and 12.9% mAP on RSNA detection at 1%. It is the right target because our model is MGCA with one component swapped. It is also the strongest method in the file that, like ours, uses no external detectors. We retrain MGCA ourselves with the same data, encoders, epochs, batch size, and a matched hyperparameter search of equal trial count. Every downstream split and seed is shared. Beating it means improving SIIM Dice and RSNA mAP at 1% by more than the seed-to-seed spread over at least three seeds, without losing CheXpert AUROC.

ASG reports 71.9% Dice on SIIM at 1% using Faster R-CNN boxes and RadGraph parsing. It is the upper reference for detector-supervised alignment. Its number likely reflects a different protocol from MGCA's, so we compare only against our own rerun under the shared protocol. We do not claim to beat it unless the rerun shows it.

For single-modality dense transport, OTCXR reports 68.8% AUC on NIH ChestX-ray14 at 1%, against 67.7% for DenseCL. We rerun it on MIMIC-CXR images to separate "transport helps" from "transport across modalities helps."

The ablations that isolate the key ingredient are all trained identically:

  • Balanced transport ($m=1$, $\rho=1$), which tests whether the discard option matters.
  • Partial transport without the patch constraint ($\rho \to \infty$), which is attention plus a discard option and tests whether capacity matters.
  • Dual-softmax matching, a single row-then-column normalization, which tests whether full Sinkhorn is needed.

Why not just…?

Why not just sharpen the attention with a lower temperature or an entropy penalty? Sharpening makes each word commit to fewer patches. It does not stop different words from committing to the same patch, because each word is still normalized independently. The greedy failure documented by From Alignment to Assignment is exactly many confident queries converging on one hub. We include a temperature-tuned GLoRIA-style term with its own search budget. If it matches our method on SIIM and RSNA and shows equally low hubness, the capacity claim is refuted.

Why not just add a "no match" slot to softmax attention? This is the most serious threat. It gives words a way to discard mass, which is half of what partial transport offers, at almost no cost. It is exactly our $\rho \to \infty$ ablation. MADPOT's gains come from discarding background, which suggests the discard option alone may carry most of the benefit. Our experiments settle it directly. If the no-match slot matches the full method on dense tasks, we would report that the discard option, not global capacity, is the operative ingredient, and recommend the cheaper method.

Why not just use detector boxes and a report parser as ASG does? Detector-supervised alignment works, and ASG's reported SIIM number is high. The cost is dependence on a frozen detector trained on typical anatomy and on a parser's vocabulary. We test whether that dependence matters on out-of-distribution transfer. We pretrain on MIMIC-CXR and evaluate on VinBig-CXR, whose 14 pathologies come from a different hospital system. If ASG's advantage holds there too, the unsupervised route is not justified on accuracy, only on pipeline simplicity.

Why not just rely on MGCA's prototype assignment, which already uses Sinkhorn? Prototype assignment operates on pooled whole-image and whole-report vectors. It cannot tell which patch a phrase refers to, so it cannot fix hubs within an image. Comparing our model against MGCA, which differs only in the local term, answers this directly.

Proposed Experiments

All models pretrain on MIMIC-CXR. We evaluate the following:

  • CheXpert classification at 1%, 10%, and 100% labels, measured by AUROC with linear probing and full fine-tuning.
  • NIH ChestX-ray14 at 1%, measured by AUC.
  • SIIM-ACR pneumothorax segmentation at 1% and 10%, measured by Dice.
  • RSNA pneumonia detection at 1% and 10%, measured by mAP.
  • Out-of-distribution transfer to VinBig-CXR.
  • Grounding: on RSNA images with boxes, we check whether the transport map of the report's finding phrase falls inside the annotated box. This pointing accuracy requires no training labels.

The decisive checks come in three parts.

First, the full method must beat our MGCA rerun on SIIM and RSNA at 1%.

Second, the gap to the no-match-slot ablation must be positive and grow as $\rho$ shrinks toward 1. It should then fall again at $\rho = 1$ exactly, where balanced transport forces background matches, giving a U-shaped curve.

Third, across every trained model, lower patch hubness should correlate with higher 1% Dice.

A secondary check targets rare findings. We report per-class CheXpert AUROC for the least frequent labels, where Self-supervised representation learning from 12-lead ECG data found self-supervised gains concentrate. Hub-free matching should help most there, because rare phrases are the ones hubs steal from.

If the first check passes but the second fails, the paper becomes a negative result on global capacity, and the cheap no-match slot becomes the recommendation.