Anchor Co-occurrence Scores for Telling Hard Examples from Noise in Web Image-Text Data

Web-scraped image-text pairs are the fuel for CLIP-style models, which are vision-language encoders trained to pull matching images and captions together. Much of that fuel is bad. Alt-text often describes a backstory rather than the picture (Generalizing Image Captions for Image-Text Parallel Corpus). The best filters therefore look beyond each pair in isolation. Two families of filters, however, give opposite advice about the same samples. Diversity-oriented pruning keeps samples far from their cluster center and throws away the "prototypical" ones, on the grounds that atypical samples carry more information (SemDeDup, Density-Based Pruning). Neighbor-consensus error detection finds that those same peripheral samples are the most likely to be mismatched pairs (LEMoN). Neither family can tell a clean but unusual pair from a noisy outlier. Both look at one modality's geometry, or at local neighbor agreement, rather than at how the two modalities co-vary across the whole corpus.

We propose a cheap, target-free score that resolves this conflict. We summarize each image by its soft assignment to a few thousand image "anchors" (representative points in embedding space) and each caption by its soft assignment to text anchors. We then count, over the whole corpus, which image anchors co-occur with which text anchors. A pair is trustworthy if its image region and its caption region co-occur more than chance would predict. We measure this with pointwise mutual information (PMI), which compares observed co-occurrence to the product of the two marginal frequencies. The score does not depend on how close the sample sits to a cluster center, and it does not depend on how many neighbors confirm it. That second property matters: raw neighbor counts penalize rare concepts, while PMI rewards a rare image region that reliably pairs with a rare text region. The whole computation costs a pass over the data times the number of anchors, never a pass over all pairs of samples.

Core Claim

Cross-modal anchor co-occurrence, weighted by PMI, separates informative hard examples from noisy ones at the periphery of concept clusters. The decisive version of the claim is a swap test. Density-Based Pruning keeps boundary samples but filters alignment with pointwise CLIPScore. If its CLIPScore stage is replaced by our score, the pruned DataComp subset should train a better CLIP at fixed compute. On injected-noise benchmarks, the gain should concentrate in the outer distance-to-centroid quantiles. Speed is a side benefit, not the claim. Exact neighbor consensus can be made fast with approximate search, so we treat that as a baseline we must beat on quality.

Research Program

Let $B^v \in \mathbb{R}^{N\times M_v}$ and $B^t \in \mathbb{R}^{N\times M_t}$ be sparse soft-membership matrices of the $N$ images and captions to $M_v$ image anchors and $M_t$ text anchors. Each row keeps its top few memberships and sums to one. The anchor co-occurrence matrix and its PMI weighting are

$$C = \tfrac{1}{N}(B^v)^\top B^t, \qquad P_{ab} = \log\frac{C_{ab}+\lambda}{p_a\, q_b + \lambda},$$

where $p_a$ and $q_b$ are the column means of $B^v$ and $B^t$, and $\lambda$ is a small smoothing constant. The score of pair $i$ is a margin computed inside the implicit $N\times N$ cross-modal matrix $S = B^v P (B^t)^\top$, which is never materialized:

$$s_i = S_{ii} - \tfrac{1}{|R_i|}\textstyle\sum_{j\in R_i} S_{ij}.$$

Here $R_i$ is a small set of randomly drawn captions. The margin asks whether caption $i$ fits image $i$ better than random captions do. Without it, generic captions that fit everything would score well.

Several pieces are borrowed. The bipartite sample-to-anchor graph comes from EMUFS and DAGE. Both cut graph cost from quadratic to $O(NM)$, but only on unimodal or multi-view data of at most about 25,000 samples. We use two anchor sets, one per modality, and couple them through the data pairing; neither paper has such a coupling. Anchor coverage of the tails is a known weak point of anchor graphs. To address it, we reuse the codebook-reassignment trick from MetaQ: rarely used anchors are moved to poorly covered points. MetaQ reports that this preserved cell populations at 0.01% frequency in single-cell data.

The motivating evidence is that neighborhood context beats pointwise similarity. LEMoN reports gains of up to 6.3% AUROC over 12 baselines on captioning error detection. Its authors also acknowledge that atypical but valid samples are penalized. Our PMI-over-anchors score is designed to keep the neighbor-context benefit without that penalty.

The constraining evidence comes from two target-aligned selectors. SDM and FLYT both select data to match ImageNet-1K and inherit its object-centric bias. We stay target-free and judge ourselves on all 38 DataComp tasks, not ImageNet alone.

The work has four stages:

  1. Build both anchor sets from one public CLIP embedding of DataComp-small and DataComp-medium.
  2. Compute $C$, $P$, and $s_i$ in streaming passes.
  3. Plug $s_i$ into a pruning pipeline in two ways: as a standalone threshold, and as the alignment stage inside Density-Based Pruning's complexity-budgeted retention.
  4. Optionally recompute anchors with the partially trained model at one or two checkpoints. This tests whether refreshing the graph helps, which the DBP and LEMoN authors list as open.

Baselines

The naive reference is DataComp's no-filtering pool together with the standard CLIPScore threshold at the same retained fraction. These run under DataComp's fixed architecture and samples-seen budget. They show whether any relational structure helps at all.

The strongest target-free competitor on the specific question is LEMoN run with approximate nearest-neighbor search. It is the method whose quality our score must match or exceed, and approximate search removes its scalability excuse. We give it the same CLIP embeddings and the same neighbor count range. Its combination weights get a hyperparameter grid of equal size to ours, selected on the same small held-out noise-injected split. On captioning error detection, the file reports LEMoN's gains only relative to baselines (up to 6.3% AUROC; flags verified correct 41% of the time versus 32.5% for CLIP similarity on Flickr30k). For DataComp-medium we found no reported result on Lacuna for LEMoN, so we run it ourselves. Beating it means higher average accuracy over the 38 tasks at equal retained fraction and equal filtering compute, with non-overlapping seed intervals.

Density-Based Pruning is the strongest diversity-oriented method and the host of our swap ablation. It reports 64.1% versus 63.0% ImageNet zero-shot on LAION with 27.7% of data and compute. For DataComp-medium we found no reported numeric result on Lacuna, so we rerun it with its DINOv2 clustering unchanged. The ablation keeps every DBP component and replaces only its CLIPScore stage with $s_i$. This isolates whether cross-modal co-occurrence is the active ingredient.

A second ablation replaces soft anchors and PMI with raw co-occurrence counts of hard k-means cluster IDs. This tests whether the smoothing and the rare-concept normalization matter.

As a target-aligned reference point, we report the published DataComp-medium ImageNet zero-shot numbers: 40.1% for M-FLYT with Soft Cap Sampling (FLYT) and 35.2% for SDM (SDM). We do not claim to beat FLYT on ImageNet. Both methods use ImageNet during selection and we do not. The comparison we care about is the average over non-ImageNet tasks, where we expect to be competitive at a small fraction of FLYT's second-order-gradient cost.

Why not just…?

Why not just run LEMoN with an approximate nearest-neighbor index? This is the real threat. Approximate search makes neighbor consensus feasible at 128M samples, so if LEMoN-with-approximate-search matches our score, our contribution reduces to a cheaper approximation. Our expected advantage is specific. LEMoN's score falls when a sample has few confirming neighbors, while PMI does not. The test is therefore stratified: detection AUROC on injected noise by distance-to-centroid decile and by anchor frequency, plus retention rates of rare-anchor samples. If LEMoN matches us in the outer deciles and on tail retention, we will report that the claim failed.

Why not just keep DBP as is, since it already filters with CLIPScore before keeping boundary samples? LEMoN shows that pointwise CLIP similarity is a weaker error detector than relational scores, and the boundary is exactly where its errors should cluster. The swap ablation answers this directly. If DBP-with-our-score does not beat DBP-with-CLIPScore, the conflict between the two families is not as costly as we argue.

Why not just recaption everything, as BLIP's CapFilt does? BLIP needs autoregressive decoding over every image plus multi-stage training. That cost dwarfs embedding-based scoring, and it replaces web context with literal captions. Recaptioning is complementary rather than competing: our score could decide which captions to rewrite. We leave that combination out of the main claim.

Why not just use the hard cluster-ID co-occurrence table? It costs the same and may suffice. The hard-assignment ablation above measures this. If it ties, we will recommend the simpler version.

Proposed Experiments

The detection experiments use the COCO and Flickr30k noise-injection protocols from LEMoN: random, category-based, and noun-based caption swaps. We measure AUROC overall and stratified by centroid-distance decile.

The pretraining experiments use DataComp-small and DataComp-medium at retained fractions of 10%, 20%, and 30%. We report ImageNet zero-shot accuracy, the VTAB average, retrieval, and the 38-task average, with three seeds each.

We also log filtering cost in GPU hours and peak memory for all filters on the same hardware.

There are two decisive checks. First, DBP-with-our-score must beat DBP-with-CLIPScore on the 38-task average at DataComp-medium. Second, on injected noise our margin over approximate-search LEMoN must be largest in the outermost deciles and for rare anchors, and near zero for central, frequent anchors. A flat improvement across deciles would mean the gain has some other cause than the one we claim. If neither check holds, the boundary-noise hypothesis is wrong.