Gradient Voting Among Caption Sources for Noisy Vision-Language Pre-Training

Image-text models such as CLIP learn by pulling each image toward its caption and pushing it away from other captions in the batch. The captions scraped from the web are often wrong or off-topic. A photo of a dog captioned "My favorite Sunday afternoon" teaches the model nothing true. The common fix is to write new captions with a large multimodal model and train on a fixed mix of web and synthetic text. That fix has two costs. First, the mix ratio is a sensitive hyperparameter. Second, synthetic captions carry their own errors: hallucinated objects, and a generic style that drops the rare names and concepts web text often contains. We propose to let the model being trained decide, image by image, which caption to believe. For each image we keep the web caption and two cheaply sampled synthetic captions, and compute how each one would move that image's embedding. Captions whose update directions agree with the others get more weight. Captions that disagree with all others get less. Images on which all three sources disagree are sent back to be recaptioned with a stronger prompt, which closes the loop between the learner and the caption generator.

In two sentences: the disagreement between the gradients that different captions induce for the same image is a cheap, learner-specific signal of caption noise. Using it to reweight captions per sample, and to decide what to regenerate, should replace a brittle global web-versus-synthetic weight. It should also beat both offline filtering and standard loss-level balancing algorithms, which assume each loss term is clean.

Core Claim

The claim has three parts. Caption noise in web-scale pre-training lives at the level of individual samples, not at the level of objectives. The learner's own per-sample gradient geometry detects it better than a frozen text-similarity model does. Once caption weights are set per sample, downstream accuracy should become nearly flat as the global mixing weight between web and synthetic losses changes. The research file supports the premise from two sides. Web noise produces destructive gradients and overconfident, miscalibrated representations (Improved baselines for vision-language pre-training, Align before Fuse). Meanwhile, the multi-task balancing methods CAGrad (Conflict-Averse Gradient Descent) and LDC-MTL (Scalable Bilevel Loss Balancing) were validated only on clean benchmarks such as NYU-v2, CelebA and Meta-World. When one loss term averages clean and corrupted samples, a per-objective weight can only scale the corruption up or down; it cannot remove it. CAGrad's rule of maximizing the worst-case task improvement may even favor the corrupted term. This is the failure we expect to demonstrate.

Mechanism

Take a batch of images with normalized embeddings $u_i$, and let $v_i^{k}$ be the text embedding of caption source $k$ for image $i$. The sources are the web caption and two synthetic captions sampled with nucleus sampling. BLIP found that sampling beats beam search for caption diversity (BLIP). For the contrastive (InfoNCE) loss with temperature $\tau$, the gradient on the image embedding when source $k$ is the positive has a closed form:

$$g_i^{k} = \frac{1}{\tau}\Big(\sum_j p_{ij}^{k}\, v_j - v_i^{k}\Big),$$

Here $p_{ij}^{k}$ is the softmax probability the model assigns to text $j$ when $v_i^k$ is the positive, and the negatives $v_j$ are shared. Computing this gradient requires no extra backward pass. Each source's weight comes from its agreement with the other sources:

$$a_i^{k} \propto \exp\Big(\kappa \cdot \tfrac{1}{2}\textstyle\sum_{k' \neq k} \cos(g_i^{k}, g_i^{k'})\Big),$$

The weights are normalized over $k$, and $\kappa$ is a sharpness constant. We show below that results are insensitive to $\kappa$ within a range. The training loss is the $a$-weighted sum of per-source contrastive losses. A web caption that alone names "Golden Gate Bridge" while the synthetic captions say "a red bridge over water" will still mostly agree in direction, so it keeps its weight and its rare concept. A web caption about Sunday afternoons will point elsewhere, so its weight drops. When all three sources pairwise conflict below a threshold, the image enters a regeneration queue. A stronger captioner then produces a new synthetic caption, prompted with the conflicting candidates. The queue is kept to a few percent of the data per epoch, so the closed loop costs far less than recaptioning everything.

Relation to Prior Work

The work that motivates this proposal documents the problem but handles it offline and globally. BLIP's CapFilt trains a captioner and a filter in separate stages before pre-training. Sieve prunes pairs whose web text is far, under a frozen sentence encoder, from a generated caption. Matched Data, Better Models selects subsets whose sparse-autoencoder concepts match a target distribution. That paper also warns that the selected subset inherits the target's blind spots. All three fix the data before training begins. We instead let the scores change as the learner's text space changes, and we reweight rather than discard.

CLIPS and FineViT supply synthetic captions at scale, but in open loop. CLIPS's finding that long synthetic captions hurt contrastive retrieval constrains our design: we use short sampled synthetic captions for the contrastive branch, not dense paragraphs. ALBEF's momentum distillation softens targets for every sample using a teacher. Our method differs because it arbitrates between competing positive captions rather than smoothing a single one.

On the task-driven synthesis side, OptimSyn supplies the core intuition we reuse: gradient alignment predicts training utility better than embedding proximity. OptimSyn applies it through reinforcement learning on a prompter for clean text fine-tuning data. We apply it per sample, inside the contrastive loss, on noisy multimodal data. Discriminability Objective for Training Descriptive Captions shows that a static weight between a utility loss and fluency collapses captions into bag-of-words. That result motivates treating global weights as the thing to eliminate, and it warns against letting the learner drive the captioner too hard. For this reason our regeneration step conditions a fixed captioner on candidates rather than training it on learner gradients.

Research Program

We will build the weighted contrastive loss and the regeneration queue on a standard open CLIP training stack. Two synthetic captions per image will be generated once with a mid-sized multimodal model. The analysis has three strands. First, we test whether gradient conflict actually tracks noise, using a subset with injected caption swaps and a small human-audited sample. Second, we measure how accuracy varies with the global web-versus-synthetic weight, with and without per-sample voting; this treats hyperparameter sensitivity as a quantity to measure, not a nuisance. Third, we study when gradient voting differs from text-embedding similarity. Early in training, the learner's text encoder is poor, and the method may behave like a noisier Sieve. We will track how the agreement between the two scores changes over training.

Baselines

The naive reference point is standard CLIP trained on raw web captions, followed by CLIP trained on a fixed 50/50 mix of web and synthetic captions. The second is the configuration most practitioners would use.

The strongest global-weight competitor is the same mix with its ratio grid-searched over five values, using the same GPU-hour budget our method gets for its own settings. It is strong because, if noise were roughly uniform, one well-chosen ratio would capture most of the gain.

The strongest existing filtering method is Sieve applied to the same pool with the same synthetic captions, at its best pruning fraction, followed by standard training. It is the closest competitor in spirit, since it also compares web text against a generated caption. No reported result was found on Lacuna for Sieve on our exact pools, so we will rerun it ourselves.

The strongest reported number in the file on a shared pool is Submodular Distribution Matching from Matched Data, Better Models: 35.2% ImageNet-1K zero-shot top-1 on DataComp-medium. At DataComp-medium scale, beating that figure under the same training budget counts as a win on classification. Because SDM uses ImageNet as its matching target, it has an advantage on ImageNet, so we will also compare on retrieval.

For loss balancing, we treat the web loss and the synthetic loss as two tasks and run LDC-MTL and CAGrad with their recommended settings. LDC-MTL is included because it is reported as nearly as fast as linear scalarization. The file reports CAGrad at a 0.20% per-task performance drop on NYU-v2, but no reported result was found on Lacuna for either method on vision-language pre-training.

Finally, three ablations isolate the key ingredient. The first replaces the learner-gradient cosine with frozen sentence-embedding cosine inside the same weighting rule. The second removes the regeneration queue. The third uses one synthetic caption instead of two.

For the method to count as beating the baselines, it must exceed the tuned-ratio baseline and Sieve on average zero-shot retrieval and ImageNet accuracy, across at least three seeds. It must also show a clearly flatter sensitivity curve.

Why not just…?

Why not just grid-search the web/synthetic ratio? If noise were spread evenly, a single ratio would work, and our method would reduce to expensive bookkeeping. The injected-noise experiment tests this directly. We vary noise so that it is concentrated in some concepts. If the tuned ratio matches our method there, the per-sample claim fails, and we will report that.

Why not just filter with Sieve and train normally? This is the most serious threat. Sieve already detects web-versus-generated mismatch, and does so cheaply. Our argument is that a frozen sentence encoder cannot see which mismatches matter to the current learner. Our argument is also that discarding loses the rare concepts that Matched Data, Better Models shows pointwise filters throw away. The ablation that swaps gradient cosine for sentence-embedding cosine settles the question. If that swap loses nothing, the contribution shrinks to soft reweighting plus regeneration, and we would say so.

Why not just apply LDC-MTL or CAGrad to the two losses? These methods choose one weight per objective per step. Web noise, though, varies within a batch, so the corrupted samples' gradients are always averaged into the web term. We expect loss-level balancing to land near the tuned-ratio baseline. If it instead matches our method, then the per-sample machinery is unnecessary.

Why not just recaption everything densely with a strong model? CLIPS reports that long synthetic captions fed to the contrastive branch reduce retrieval. Dense recaptioning at FineViT's scale also costs far more than our regeneration of a few percent of images. We include full recaptioning with the strong captioner on the smaller pool as a reference, to show what the queue's compute saving costs in accuracy.

Proposed Experiments

The main runs pre-train ViT-B/16 CLIP models on CC12M, which ALBEF also used. We then scale to DataComp-medium's 128M pairs for the comparison against SDM. Evaluation covers zero-shot image and text retrieval on MSCOCO and Flickr30K and zero-shot ImageNet-1K top-1. We will also report expected calibration error on retrieval, since miscalibration is the documented symptom of web noise.

The first decisive check injects synthetic noise into CC12M by swapping captions for a known 20% of images, concentrated in chosen classes. We then measure how well each score (gradient conflict, Sieve distance, CLIPScore) recovers the swapped pairs, as area under the ROC curve at several points during training. The method should overtake Sieve after early training, as the learner's text space matures.

The second check is the sensitivity curve. We plot retrieval and ImageNet accuracy against the global web/synthetic weight, from 0.1 to 0.9, for fixed mixing, LDC-MTL, and our method. The prediction is a near-flat curve for ours, and a peaked curve for the others.

The third check is rare-concept retention. We measure zero-shot accuracy on the least frequent ImageNet classes and on named-entity retrieval queries drawn from COCO captions. Here we expect reweighting to beat Sieve's pruning.

The method is refuted if, at equal compute, it fails to beat both the tuned ratio and Sieve on the headline metrics. It is also refuted if its sensitivity curve is no flatter than theirs.