Image-text models such as CLIP learn by pulling each image toward its own caption and pushing it away from other captions. They get much sharper when the "other" captions are hard ones: captions the model currently finds convincing but that describe something else. The classic way to find such examples is iterative hard negative mining. You train a model, run it over a large pool, collect its confident mistakes, and retrain on them. This worked for decades in face and pedestrian detection. On web-scraped data it breaks down. The model's most confident "mistakes" are often correct matches that the web simply never labeled as matches: a different photo of boats in a marina, or a caption written for a near-duplicate image. Penalizing those teaches the model to forget real concepts.
We propose to run genuine multi-round, corpus-level hard negative mining for dual-encoder image-text pretraining. The pool is refreshed from the whole training corpus every round, not just the current batch. We separate true hard negatives from unlabeled true matches with a signal the iteration itself provides. A real hard negative such as a wolf photo for a "husky" caption should move away once it is penalized. An unlabeled true match should resist, because many other pairs in the corpus keep pulling it back. We call this resistance and measure it between rounds. Combined with a check of whether the corpus's own nearest neighbors vouch for the pairing, it decides whether each mined candidate is penalized, promoted to a soft positive, or dropped. No external judge model is needed.
Core Claim
Our main claim is that a candidate's response to being penalized carries information about whether it is truly negative, and that this information is cheap to collect inside an iterative mining loop. Suppose candidate caption $t'$ for image $x$ was used as a negative during round $r$. Let $s_r(x,t')$ be its cosine similarity under the round-$r$ checkpoint. Let $\bar{\Delta}_r(s)$ be the average similarity drop over all penalized negatives that started at a similar similarity $s$. The resistance of the candidate is then
$$\rho(x,t') = 1 - \frac{s_r(x,t') - s_{r+1}(x,t')}{\bar{\Delta}_r\big(s_r(x,t')\big)}.$$
A value near 0 means the candidate fell as fast as typical negatives. A value near 1 or above means it did not move or bounced back. Normalizing by negatives of similar starting hardness prevents resistance from simply measuring how hard a candidate was to begin with.
We hypothesize that high resistance identifies false negatives. The reason is that penalizing a true match conflicts with gradients from the many corpus pairs expressing the same concept, while a genuine hard negative has no such support. The hypothesis can fail in a specific way: the model may also resist on spurious web correlations it has firmly memorized. We therefore pair resistance with a corpus-consensus score in the style of LEMoN, which flags mislabeled pairs by checking whether an image's nearest image neighbors agree with a caption and vice versa. In our use, the consensus score asks whether $x$'s image neighbors are paired with captions close to $t'$.
The two signals combine as follows. Candidates with high resistance and high consensus become soft positives. Candidates with high resistance and low consensus are dropped as ambiguous. Low-resistance candidates stay as weighted hard negatives.
Relation to Prior Work
The motivating evidence comes from MAFA and its earlier version, Converting and Smoothing False Negatives. They show that hard-negative batches push the false-negative rate from about 3% to over 20%. They also show that managing false negatives matters more than aggressively cleaning noisy captions. MAFA's remedy is a static, pretrained connection discriminator that promotes suspected false negatives, plus uniform label smoothing, all applied within batches. We drop the external discriminator and replace per-batch screening with a corpus-level loop whose own dynamics do the screening.
FALCON learns a per-anchor hardness quantile, rewarded by reductions in masked-language-modeling loss. It reports gains of 2.4% to 11.9% on noisy DataComp-1M. FALCON still searches only within the mini-batch, and it needs a generative head that a plain CLIP-style dual encoder lacks. Our method works with a dual encoder and searches the whole corpus.
Three other methods also bring hard negatives into vision-language pretraining. DiHT up-weights hard in-batch negatives by importance sampling after rule-based filtering. ALBEF mines from a memory queue for its matching head and softens contrastive targets with a moving-average teacher. TripletCLIP synthesizes negative captions and images offline. None of these audits the full corpus repeatedly as the model changes, which the classical bootstrapping literature found decisive. That literature includes Rowley et al., who found that iteratively collected non-face patches mattered more than adding random background, and Sermanet et al., who found that capacity did not replace bootstrapping. In retrieval, Saving Dense Retriever from Shortcut Dependency and Dense Hierarchical Retrieval mine the model's own top errors and gain from iterating. The latter reports top-1 NQ accuracy rising to 57.0% with iterative training.
TripletCLIP also constrains our design. Its ablation shows that freezing the vision encoder removes compositional gains, so the negatives we mine must update the image side. Our mined candidates therefore include hard negative images for each caption, not just hard captions.
Static filters supply our pool and serve as baselines. These include DataComp's CLIPScore filtering and Sieve, which compares generated captions to alt-text. Sieve's critique that filters built on a model's own embeddings are circular applies to us too. That critique is why consensus is computed over corpus neighborhoods rather than a single pair score, and why we test against a planted ground truth.
Research Program
Training proceeds in R rounds, initially four, of a fixed-step contrastive schedule on a DataComp pool. At the end of each round we freeze the checkpoint, embed the full pool, and build an approximate nearest-neighbor index. For every pair we retrieve its top-K non-paired captions and top-K non-paired images.
During the next round, each anchor's contrastive loss adds its mined candidates alongside in-batch negatives. Low-resistance candidates are weighted as hard negatives. Soft positives receive a share of target probability proportional to their consensus score; this is instance-adaptive smoothing, in contrast to MAFA's uniform smoothing. Dropped candidates are excluded. Candidates new to the pool have no resistance history yet. They enter at a small, conservative negative weight, so that their first round of exposure yields the measurement.
We will analyze three things. First, how the false-negative fraction among mined candidates evolves across rounds, to replicate and extend MAFA's 3%→20% finding to corpus-level mining. Second, whether resistance separates false from true negatives better than raw similarity or consensus alone. Third, how mining compute trades against training compute.
Baselines
The naive reference is standard CLIP contrastive training with random in-batch negatives on the same filtered DataComp pool, using the DataComp CLIPScore filter. It shows how much any form of mining adds.
The strongest static-curation competitor is Sieve applied to the same pool, followed by plain training. Sieve reports beating CLIPScore filtering by 2.6% at the 128M medium scale averaged over 38 tasks (Sieve). The file has no absolute DataComp-medium average for it, so we report no reported result found on Lacuna for the absolute Sieve medium-scale score, and we rerun it ourselves.
The strongest mining competitors are three reimplementations on the identical dual encoder and pool. The first is DiHT's in-batch hard-negative loss. The second is MAFA-style screening, with its connection discriminator applied to our globally mined candidates. The third is FALCON's quantile scheduler, with its reward adapted for a dual encoder. For reference, MAFA reports 78.0% image-retrieval Recall@1 on MS-COCO versus 76.6% for GRIT-VLP, and FALCON reports 62.28 Recall@1 versus 55.60 on MS-COCO retrieval. Both numbers come from different architectures and pretraining sets, so they are not directly comparable. They motivate rerunning rather than quoting.
Fairness is enforced by total compute. The cost of embedding and indexing the pool each round is added to our budget. Every baseline gets the equivalent compute as extra training steps, plus the same hyperparameter-search budget. MAFA's discriminator inference is counted against its budget.
Two ablations isolate the key ingredient. "Global mining, no screening" penalizes all mined candidates. "Global mining, consensus-only screening" removes resistance. Beating the baselines means improving the DataComp 38-task average and COCO/Flickr retrieval over the best compute-matched competitor. The margin must exceed seed variance, measured over three seeds at the small scale. It also means not regressing ImageNet zero-shot accuracy.
Why not just…?
Why not just use MAFA's discriminator on globally mined candidates? This is the most serious threat, and it is run as a baseline. The case against it is that the discriminator is fixed while the candidate distribution shifts every round. The planted-false-negative test below measures directly whether its precision decays across rounds while resistance stays stable. If it does not decay, the honest conclusion is that resistance adds nothing beyond a cheaper substitute for the discriminator, and we would report it that way.
Why not just skip the top-ranked candidates and mine from a band, say ranks 20–200? This common retrieval heuristic avoids most false negatives at no cost and is a real competitor. Its weakness is that it also discards the hardest true negatives, which classical bootstrapping found most valuable. We tune the band on the same budget. If band mining matches our method on retrieval and SugarCrepe, the resistance signal is not earning its keep.
Why not just smooth labels or use momentum soft targets and skip mining? MAFA found smoothing alone insufficient without explicit mining. ALBEF still needed queue-based mining for discriminative matching. We include uniform smoothing with random negatives as a baseline to confirm this at our scale.
Why not just filter harder or scale the data? DataComp and DiHT show that curated subsets beat raw volume. Winoground reports Group Scores of 14.50% and 14.25% for strong models, below the 16.67% chance level, which shows that scale does not fix fine-grained confusions. Static filtering also cannot react to the confusions a model develops mid-training. The Sieve baseline tests this directly.
Proposed Experiments
Development and ablations use DataComp small, with the main comparison on DataComp medium (128M). Evaluation covers the DataComp 38-task suite, ImageNet zero-shot, MS-COCO and Flickr30K retrieval, SugarCrepe, and Winoground. SugarCrepe's adversarial filtering of text-only shortcuts matters here, because it prevents gains that come from caption artifacts.
The first decisive check uses a planted ground truth. Into the pool we inject known false negatives: near-duplicate images with independently paraphrased captions, stored as separate unpaired entries. We then measure, per round, the area under the precision-recall curve for recovering them with raw similarity, consensus, MAFA's discriminator, and resistance-plus-consensus. Resistance must beat consensus alone, or the core claim fails.
The second check is a human audit of 500 mined top candidates per round. It estimates the true false-negative rate, testing whether corpus-level mining exceeds MAFA's 20%. It also measures what fraction our screening catches.
The third check tests whether resistance is merely memorized spurious correlation. We compare resistance on OCR-dominated pairs, a known shortcut that DiHT's text-spotting filter targets, against other pairs. If text-heavy pairs resist disproportionately without consensus support, the drop rule is doing necessary work. If they resist with consensus support, the method has the circularity problem Sieve warns about.
Finally, plotting downstream accuracy against the number of rounds at fixed total compute shows whether iteration, rather than a single global mining pass, is what helps.
References
- LEMoN: Label Error Detection using Multimodal Neighbors
- MAFA: Managing False Negatives for Vision-Language Pre-Training
- Converting and Smoothing False Negatives for Vision-Language Pre-training
- FALCON: False-Negative Aware Learning of Contrastive Negatives in Vision-Language Pretraining
- Filtering, Distillation, and Hard Negatives for Vision-Language Pre-Training
- Align before Fuse: Vision and Language Representation Learning with Momentum Distillation
- TripletCLIP: Improving Compositional Reasoning of CLIP via Synthetic Vision-Language Negatives
- Human Face Detection in Visual Scenes
- Pedestrian Detection with Unsupervised Multi-stage Feature Learning
- Saving Dense Retriever from Shortcut Dependency in Conversational Search
- Dense Hierarchical Retrieval for Open-domain Question Answering
- DataComp: In search of the next generation of multimodal datasets
- SIEVE: Multimodal Dataset Pruning Using Image Captioning Models
- Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality
- SugarCrepe: Fixing Hackable Benchmarks for Vision-Language Compositionality