Minimal-Pair Negatives for Radiology Image–Report Contrastive Learning

Models that learn to match chest X-rays with their radiology reports are trained with a contrastive objective. The model pulls an image toward its own report and pushes it away from other reports in the same training batch. The reports it pushes away are mostly easy to tell apart, because they come from other patients, other findings and other writing styles. The model can therefore score well without ever learning what radiologists care about most: whether a finding is present, absent or hedged ("no effusion" versus "small effusion" versus "cannot exclude effusion"), and on which side it appears. We propose to build the hard contrasts directly. For every training report, we make controlled one-finding edits: flip a negation, swap an uncertain mention to a definite one, swap left and right, or substitute a sibling finding. A clinical labeler checks each edit. The edited reports serve as the negatives that the image must reject in favor of its true report.

The opportunity sits between three lines of work that have not been joined. Hard negative mining shows that uninformative negatives stall contrastive training. The medical contrastive literature shows that the usual fix, picking the most similar non-matching samples, pulls in false negatives, because many patients share the same diagnosis. The modality-gap literature shows that image and text embeddings sit in separate regions of the shared space, and that repulsion from negatives can widen that separation. Edit-built negatives are guaranteed to differ in meaning, so they avoid the false-negative trap by construction. Whether they also avoid widening the gap is an open, measurable question that this work answers.

Core Claim

In radiology image–report pre-training, the most useful negatives are verified minimal pairs of the positive report. These negatives differ in exactly one clinical finding's presence, certainty or location. Such negatives should teach the image encoder finding-level distinctions that neither similarity-based mining nor semantic soft-labeling provides. We expect large gains on negation-, hedge- and laterality-sensitive matching and no loss on standard zero-shot classification.

A second, riskier claim concerns geometry. Edit negatives live in the text region right next to the true report. Their repulsion should therefore act along finding directions inside the text region rather than as a global push between the image and text regions. If so, they sharpen the model without enlarging the modality gap. Similarity-mined negatives, by contrast, may enlarge it. Domain-specific pre-training enters as a precondition, not as the main contribution. A text encoder that has not learned clinical negation and hedging during pre-training may be unable to separate the edits at all. We test this directly.

Relation to Prior Work

Two papers supply the problem and one supplies the construction idea. ConVIRT trains a ResNet50 and ClinicalBERT dual encoder on MIMIC-CXR pairs, using ordinary in-batch negatives. It is our base architecture and our weakest contrastive reference. MedCLIP identifies the false-negative problem: patients who share a pathology get pushed apart. It replaces binary targets with soft similarity based on overlap of concepts from the Unified Medical Language System. MedCLIP softens negatives but never builds harder ones, and the file records that it keeps an anisotropic embedding space. We reuse its soft targets for ordinary in-batch reports and add edit negatives on top.

Contrastive Learning for Weakly Supervised Phrase Grounding supplies the construction idea. It builds contextually similar but factually altered captions so that a general-domain model grounds specific words. We carry this over to clinical reports, and we add a verification step that the natural-caption setting did not need. Augmenting Legal Judgment Prediction with Contrastive Case Relations makes the same point in a unimodal specialized domain: surface-similar near-misses force substantive reasoning.

The hard-negative literature motivates the method and also constrains it. Adversarial Contrastive Estimation explains why easy negatives contribute vanishing gradient. MoCHi synthesizes hard negatives by mixing embedding vectors. Contrastive Learning with Hard Negative Samples reweights negatives toward hardness and debiases false negatives using augmentations. Both of the latter two work in a single modality. Mixing vectors across modalities would place synthetic points in the empty region between image and text embeddings, which corresponds to no real image or report. Augmentation-based debiasing cannot express cross-modal clinical semantics. Our negatives are real, readable reports whose semantics are fixed by construction.

The modality-gap papers constrain what we must measure. Mind the Gap shows that ConVIRT itself has a persistent gap. It also shows that the gap's size behaves like a hyperparameter that can either help or hurt accuracy and fairness. Explaining and Mitigating the Modality Gap attributes the gap's initial growth to mismatched negative pairs. It shows that the gap closes only very slowly unless one intervenes, for example by scheduling the temperature. Its Not a Modality Gap closes the gap with alignment and uniformity losses. Closing the Modality Gap for Mixed Modality Search closes it after training by subtracting each modality's mean embedding (GR-CLIP).

None of these papers changes which negatives are used. Taken together, they warn that adding stronger negatives could enlarge the gap. That is why gap geometry is a measured outcome in our design rather than an afterthought.

Research Program

Building the edits. We run the rule-based CheXpert labeler over each MIMIC-CXR report. The CheXpert paper reports a micro-average F1 of 0.969 for this labeler. It tells us which of 14 findings are mentioned in the report and whether each is positive, negative or uncertain. For each mention, rule-based rewriting produces candidate edits of four kinds:

  • negation flips;
  • certainty changes, between uncertain and positive or between uncertain and negative;
  • laterality swaps;
  • substitution with a sibling finding in the same anatomical region.

An edit is kept only if re-labeling shows that exactly the intended entry of the label vector changed.

Removing edit artifacts. A model could learn to spot the traces of editing instead of the clinical meaning. To prevent this, we also generate meaning-preserving edits with the same templates, such as paraphrasing a negative mention in another negation form. These are treated as positives. The text encoder therefore cannot win by detecting "edited-ness."

Objective. Let $v_i$ be the image embedding, $t_i$ the paired report, $\{t_j\}$ the other in-batch reports, and $\{\tilde t_{ik}\}_{k=1}^{K}$ the $K$ verified edit negatives of report $i$. Let $w_{ij}$ be MedCLIP-style soft targets derived from label overlap. The image-to-text loss is a soft-target InfoNCE whose denominator adds the edit negatives:

$$ \mathcal{L}_i = -\sum_j w_{ij}\log\frac{e^{s(v_i,t_j)/\tau}}{\sum_{j'} e^{s(v_i,t_{j'})/\tau}+\lambda\sum_{k} e^{s(v_i,\tilde t_{ik})/\tau}} $$

Here $s$ is cosine similarity, $\tau$ is the temperature and $\lambda$ weights the edit negatives. The text-to-image direction keeps the standard form, because edits exist only on the text side.

Measurements. For every trained model we record:

  • the centroid distance between modalities, following Mind the Gap;
  • similarity between edit pairs in the text space alone;
  • how the gap evolves over the course of training, to check whether edit negatives change the early growth phase described by Yaras et al.

Baselines

The naive reference point is ConVIRT-style training with uniform in-batch negatives. It uses our exact backbones: ResNet50 for images and ClinicalBERT for text. The file's reported number for ConVIRT is RSNA Pneumonia linear-probe AUC of 90.7 with 1% labels. We retrain it ourselves on the same MIMIC-CXR split, for the same number of epochs, with the same batch size, so that it is the like-for-like floor.

The strongest existing method is MedCLIP's knowledge-driven soft-target loss. On CheXpert-5x200, MedCLIP reports zero-shot accuracy of 0.594, against 0.432 for GLoRIA and 0.418 for ConVIRT. MedCLIP pre-trained on MIMIC-CXR plus CheXpert with a ViT image encoder. We therefore treat 0.594 as an external reference and reproduce MedCLIP's loss on our MIMIC-CXR-only data with both backbones for the controlled comparison.

The key ablation is label-filtered similarity mining. It uses Robinson et al.'s hardness reweighting, but restricts the negative pool to reports whose CheXpert label vector differs from the anchor's. It uses the same soft targets, compute and tuning grid as our method. This isolates the proposal's actual ingredient: minimal-pair construction versus choosing hard-but-verified negatives from real reports.

We also run two gap-mitigation baselines: temperature scheduling from Yaras et al., and GR-CLIP centering applied after training. Each is run alone and combined with our method, so we can tell whether any gains come from closing the gap or from the negatives.

A last ablation keeps everything fixed but swaps ClinicalBERT for a general-domain BERT. It tests whether domain pre-training is what makes the edits learnable.

Fairness of comparison. Every method gets the same number of hyperparameter configurations, a grid of 12 of which the best on a validation split is kept. Every method gets the same total gradient steps. Every method sees the same number of text encodings per step. We lower the in-batch report count for our method to offset the $K$ extra edit encodings.

What counts as beating the baselines. Our method must clearly exceed the reproduced MedCLIP and label-filtered mining on the minimal-pair test described below. It must also match or exceed them on CheXpert-5x200 and RSNA, with confidence intervals from three seeds.

Why not just…?

Why not just use MedCLIP's soft targets? Soft targets stop the model from punishing shared diagnoses, but they only reweight negatives that were already in the batch. Those negatives rarely differ from the anchor by a single negation or side. MedCLIP's own authors note residual anisotropy and prompt sensitivity. The file also records that its concept extractor struggles with negation and hedging, the very distinctions our edits target. The minimal-pair test settles this directly, since MedCLIP is reproduced under identical data.

Why not just mine the most similar real reports that have different labels? This is the real threat. It is simpler, uses no rewriting, and already avoids false negatives. Our argument against it is that the nearest real report with different labels usually differs in patient, style and several findings at once. The model can then separate it without isolating the one finding that matters. That argument could be wrong. If label-filtered mining reaches our method's minimal-pair accuracy within noise, we will report that the edits are unnecessary and recommend the simpler method. The ablation is designed to make that outcome visible.

Why not just close the modality gap with temperature scheduling or mean-centering? Those methods move whole clusters or change the strength of repulsion. They do not teach the model that "no effusion" and "small effusion" must match different images. The combined runs test this. If gap closure alone recovers the minimal-pair gains, our claim fails. Mind the Gap also warns that fully closing the gap can hurt zero-shot accuracy and fairness, so gap closure is not a free improvement in this setting either.

Why not just use bigger batches? Bigger batches of random reports contain more easy negatives and more patients with the same diagnosis. The first effect adds little signal, as MoCHi and Robinson et al. show. The second makes the false-negative problem worse, as MedCLIP argues. We include one run of the ConVIRT baseline at double batch size as a check.

Proposed Experiments and Decisive Checks

Pre-training data. All models pre-train on MIMIC-CXR image–report pairs, holding out reports by patient.

Standard evaluations. We report zero-shot accuracy on CheXpert-5x200. We report text-to-image Prec@5 on CheXpert-8x200, where ConVIRT reports 60.0. We report RSNA Pneumonia AUC from a linear probe trained on 1% and on 10% of labels.

The primary new measure. The main measure is a minimal-pair matching test. It is built from held-out MIMIC-CXR studies whose edits we sample for radiologist verification. For each image, the model must rank its true report above every verified edit of it. Accuracy is reported separately for negation, certainty, laterality and sibling-finding edits. No reported result exists on Lacuna for this test, so every baseline is run by us.

Geometry and fairness. Gap measurements come from the centroid distances and training curves described above. Fairness is evaluated with the clip_cxr_fairness codebase on MIMIC-CXR and NIH-CXR14 subgroups. Because Mind the Gap ties gap size to subgroup harms, any change in the gap must be checked for fairness side effects. No reported numbers for these subgroup evaluations were found on Lacuna.

Decisive checks. Three outcomes decide the project.

First, the method must beat label-filtered mining on minimal-pair accuracy, especially on negation and certainty edits. It must do so without losing CheXpert-5x200 accuracy. If it does not, the construction idea is refuted.

Second, the gains must survive the addition of temperature scheduling. They must also not be reproduced by gap closure alone. Otherwise we have only found an indirect way to close the gap.

Third, the general-BERT ablation must show a markedly smaller benefit from edits. If it does not, domain pre-training is not the enabling factor we claim, and we will say so.

The geometric prediction is judged separately and can fail without sinking the main claim. The prediction is that edit negatives leave the centroid gap unchanged while similarity mining enlarges it. If instead edit negatives enlarge the gap, the paired runs with temperature scheduling tell us whether the gap costs accuracy, fairness, or neither in this domain.