Medical vision-language models learn by matching chest X-rays to radiology reports. Two mismatches are built into that data. Some diseases are far rarer than others. And a report describes only part of what an image shows: radiologists mention the findings they judge relevant and leave the rest unsaid. Current training objectives handle both mismatches badly. They require every image to spread all of its "matching weight" across the texts in a batch, so a normal-looking scan is pushed toward whatever reports happen to be present, and a scan with an unreported finding is matched as if its report were complete. We propose replacing the matching targets with the solution of an unbalanced optimal transport problem. Optimal transport is a way of pairing two collections at minimum total cost. The unbalanced version is allowed to leave some weight unpaired, at a price. This lets the training signal say "this image has content that no text in this batch describes" rather than forcing a wrong match.
The bet is specific and falsifiable. If the reasoning holds, the model should recognize rare pathologies better. It should also stop using the geometric separation between image and text embeddings (the "modality gap") as its only outlet for unmatched information. Discarded transport mass would take over part of that role. Because transport is solved only over a mini-batch, the cost is a few Sinkhorn iterations on a 128×128 matrix. That keeps the method within the compute budget of a compact, domain-specialized model rather than a frontier system.
Core Claim
During domain-specialized contrastive training, the soft image-to-text target matrix should come from a batch-level unbalanced transport plan. Its text-side marginals should be weighted by how much each report actually says, and its marginal constraints should be relaxed. A marginal here is the total matching weight each item must send or receive. Fixed per-row normalized targets should no longer be used. The claim rests on three findings from separate literatures that have not been put together.
The first is Two Effects, One Trigger. It shows that the modality gap is driven by information imbalance: captions that omit visual attributes widen the gap, which then acts as an entropy regulator. The same paper shows that closing the gap after training hurts downstream performance. That result constrains our design. We do not try to shrink the gap. We give the model a different outlet for the missing information.
The second is MedCLIP. It decouples images and reports into separate pools and builds soft targets from dot products of medical concept vectors extracted by MetaMap from the Unified Medical Language System (UMLS). This fixes the false-negative problem, where two patients with the same disease are pushed apart. But every image's target row still sums to one, and nothing accounts for the long tail of pathologies. MedCLIP supplies our base architecture and our semantic cost.
The third line is the imbalance-sensitive transport work: ROOT, JCPOT, and COT. Together they show that strict mass conservation forces majority-class mass into the wrong categories when class frequencies differ. ROOT further proves that relaxed marginals suppress mismatched samples exponentially in their transport cost. These papers motivate the relaxation and supply the solver form.
The closest existing work also uses optimal transport in vision-language settings, and we differ from each in a concrete way. UOT-RCL aligns relational geometry across modalities with Sinkhorn under symmetric label noise on generic retrieval benchmarks. NLPrompt uses transport to text prototypes to split clean from noisy samples for prompt tuning of a frozen CLIP. Both use mass-conserving transport, both assume roughly balanced class support, and neither ties the marginals to report information content.
Research Program
For a batch of $n$ images and $m$ reports, let $C_{ij}$ be a cost with two parts. The first is the UMLS concept disagreement between image $i$'s labels and report $j$'s extracted concepts, as in MedCLIP. The second is $\lambda$ times the current, gradient-detached embedding distance. We compute the target plan
$$\gamma^\star = \arg\min_{\gamma \ge 0} \langle C, \gamma\rangle - \varepsilon H(\gamma) + \tau\,\mathrm{KL}(\gamma\mathbf{1}\,\|\,a) + \tau\,\mathrm{KL}(\gamma^\top\mathbf{1}\,\|\,b).$$
The symbols are as follows. $H$ is the entropy of the plan. $\varepsilon$ is the entropic smoothing. $\tau$ is the price for leaving mass unmatched; as $\tau\to\infty$ the problem returns to balanced transport. $a$ is uniform over images. $b_j$ is proportional to the number of distinct findings, positive or negated, that report $j$ asserts.
The model's image-to-text and text-to-image similarity matrices are then trained to match $\gamma^\star$ with a weighted cross-entropy. The weight on row $i$ is its transported mass $(\gamma^\star\mathbf{1})_i$. This means images whose content no batch report describes contribute little gradient, so they are not pulled toward a wrong text.
The weighted text marginal is what links the two kinds of imbalance. A report that names three findings can absorb more image mass than a terse "no acute process." A rare-pathology report can absorb mass only from images that share its concepts, because pushing normal-scan mass onto it costs more than discarding that mass.
Domain specialization sets the regime and the constraint. We follow MedCLIP's backbone and its MIMIC-CXR plus CheXpert pretraining, at the same scale and the same data budget. We also test a parameter-efficient variant. In that variant both encoders are frozen and only a bridge is trained: a linear layer followed by a small MLP, the split that Ex-MCR found preserves intra-modal structure. That variant asks whether the targets alone, without encoder retraining, carry the benefit.
Before any medical work, we run the whole method on the synthetic MAD benchmark from Two Effects, One Trigger. MAD lets us control caption thinning and class frequencies independently and gives ground truth for which image attributes went unreported.
Baselines
The naive reference is MedCLIP's own loss, trained with our pipeline on identical data, augmentation, epochs and seeds. The published reference point is MedCLIP-ViT's zero-shot accuracy of 0.594 on CheXpert-5x200, against 0.432 for GLoRIA and 0.418 for ConVIRT, as reported in MedCLIP. That paper also reports a fine-tuned accuracy of 0.807 on RSNA Pneumonia. We first reproduce MedCLIP within our compute budget; all claims compare against our reproduction, with the published 0.594 as a sanity check.
The strongest imbalance-aware competitor is MedCLIP's loss combined with class-balanced batch sampling over CheXpert labels and logit adjustment by the training-label prior. This is the cheap, standard remedy a practitioner would try first, so it is the baseline we must beat. It gets the same tuning budget as our method: an equal number of trials over its sampling temperature and adjustment strength.
The strongest transport competitor is balanced Sinkhorn targets with the same cost. This mirrors the mass-conserving transport in UOT-RCL and PromptOT, adapted to our setting, since no published number exists for either on chest X-rays. It is also the $\tau\to\infty$ ablation of our method.
Two further ablations isolate the key ingredients. One sets the text marginal $b$ to uniform, removing the information-content weighting. The other drops the geometric term ($\lambda=0$), testing whether the plan needs the evolving embedding at all.
Beating the field means three things on a long-tailed evaluation, all with seed-level confidence intervals that exclude zero. First, higher macro recall on the rarest quartile of pathologies than balanced sampling with logit adjustment. Second, no loss of CheXpert-5x200 accuracy relative to our MedCLIP reproduction. Third, lower expected calibration error (ECE).
For the long-tailed setting we found no reported result on Lacuna for MedCLIP or any other method. For MAD, no reported result found on Lacuna for retrieval or gap numbers beyond qualitative trends.
Why not just…?
Why not just use class-balanced sampling or a class-weighted loss? This is the real threat. If balanced sampling matches our rare-class recall, the contribution reduces to a sampling trick, and we will report it that way. Our reason for expecting a difference is that reweighting acts on class frequency, not on what a report leaves out. It cannot stop an image with an unreported finding from being matched as if its report were complete. On MAD we can raise caption thinning while holding class balance fixed. If balanced sampling and our method stay tied as thinning increases, the information-marginal part of our idea is unsupported. WDAN shows that static weights fail when priors shift between domains. That is suggestive but not decisive here, so the experiment carries the argument.
Why not just correct for prevalence at inference time by re-estimating class priors on the target hospital's images? We first considered a test-time unbalanced transport method. We abandoned it because expectation-maximization prior correction on temperature-calibrated zero-shot scores would plausibly do as well, and it costs nothing. We keep that correction as an add-on applied to every trained model. The question is whether training-time targets still help after it is applied. If the gains disappear once all models receive prior correction, our method is only fixing a prior mismatch, and we will say so.
Why not just close the modality gap by subtracting the mean image-text offset? Two Effects, One Trigger shows that this post-hoc closure damages neighborhood structure and downstream accuracy. We include it only as a diagnostic on MAD. Our hypothesis is about where unmatched information goes, not about gap size, and we measure gap size so the hypothesis can fail.
Why not just filter low-confidence or low-similarity pairs with a threshold? AgriPerceiver filtered teacher labels at confidence 0.5 and still showed an ECE of 0.139. ROOT argues that any fixed retention quota either admits harmful samples or discards rare ones. We test a similarity-threshold variant tuned on validation data. If it matches our method, the transport machinery is unnecessary.
Proposed Experiments
The first experiment is a factorial on MAD. One axis is class imbalance, with minority classes at 50%, 10% and 2%. The other is caption thinning, from 0% to 75% of attributes omitted. We measure retrieval, per-class recall, and the Relative Modality Gap (RMG) metric. The predictions are that balanced transport degrades as imbalance grows while our method does not, and that RMG grows more slowly with thinning under our method at equal retrieval.
The decisive mechanistic check is also on MAD, where we know exactly which image attributes each caption omits. The per-image discarded mass, $1-(\gamma^\star\mathbf{1})_i/a_i$, should correlate with the number of omitted attributes. If it does not, the mechanism we describe is not what drives any gains.
The medical experiments pretrain on MIMIC-CXR and CheXpert. We evaluate zero-shot on CheXpert-5x200 and RSNA Pneumonia. We also evaluate on prevalence-shifted resamplings of the CheXpert test studies, which we build so that pathologies such as edema or consolidation occur at between 1% and 30%. Metrics are balanced accuracy, rarest-quartile recall, macro AUC, ECE, and RMG. Everything is repeated for the frozen-encoder bridge variant to check that the effect does not depend on encoder retraining.
The result that would settle the proposal against it is this. After every model receives the same inference-time prior correction, balanced sampling with logit adjustment matches our method on long-tailed rare-class recall, and the uniform-$b$ ablation matches the full method on thinned MAD captions. That outcome would show that neither relaxed marginals nor information-weighted marginals add anything beyond standard reweighting.
References
- Two Effects, One Trigger: On the Modality Gap, Object Bias, and Information Imbalance in Contrastive Vision-Language Models
- MedCLIP: Contrastive Learning from Unpaired Medical Images and Text
- Transport or Discard: Robust Unbalanced Optimal Transport for Cross-Domain Policy Adaptation
- Optimal Transport for Multi-source Domain Adaptation under Target Shift
- COT: Unsupervised Domain Adaptation with Clustering and Optimal Transport
- A Unified Optimal Transport Framework for Cross-Modal Retrieval with Noisy Labels
- NLPrompt: Noise-Label Prompt Learning for Vision-Language Models
- Extending Multi-modal Contrastive Representations
- Mind the Class Weight Bias: Weighted Maximum Mean Discrepancy for Unsupervised Domain Adaptation
- AgriPerceiver: A Parameter-Efficient Vision-Language Model for Structured Macroscopic Crop Phenotyping