Filtering Web Image-Text Data by Comparing Neighborhoods Within Each Modality

Web-scraped image-text pairs are cheap and plentiful, and a large fraction of them are wrong. An alt-text such as "my favorite Sunday afternoon" attached to a photo of a dog teaches a contrastive model to link fur and grass with the word "Sunday." The standard fix is to score every pair with a pretrained image-text model such as CLIP and keep the high scorers. This has two problems. The filter inherits the biases of the model it is meant to improve, a circularity that SIEVE points out. And it depends on comparing an image vector directly to a text vector across the "modality gap," the empty region that separates image and text clusters in a shared embedding space. The main alternative is to caption every image with a generative model. That breaks the circularity, but on billions of images its cost starts to rival pre-training.

We propose a filter that never compares an image to a text. For each pair $i$, we find its nearest neighbors among the images using a frozen image-only encoder, and separately its nearest neighbors among the texts using a frozen text-only encoder. We then ask whether the two neighbor sets overlap. If a pair is correct, pairs with similar images should tend to have similar captions, so the two neighborhoods should share many members. If a caption is generic or unrelated, its text neighbors will be other generic captions whose images look like anything, and the overlap falls to chance. The score comes entirely from within-modality structure, so it avoids the modality gap and uses no image-text model at all. Because every neighborhood can be computed offline, one sample at a time, against a prebuilt nearest-neighbor index, the method runs at roughly the cost of CLIPScore and far below the cost of captioning. New data can be scored by index lookup alone.

Core Claim

Agreement between within-modality neighborhoods is a noise signal that is independent of any image-text model. It is at least as good as CLIPScore for curating pre-training data, and it adds to CLIPScore when the two are combined. It matches or beats neighborhood methods that operate inside a shared CLIP space, at the compute cost of a single forward pass per modality. The new element is the "relational" comparison. We compare the structure of two separate spaces and never measure distances inside a joint space.

Relation to Prior Work

Three papers motivate this design. LEMoN shows that neighborhood consensus finds label errors better than isolated pairwise similarity. Its discordance score combines the direct image-text distance with image-to-text and text-to-image neighborhood distances, all measured in a CLIP space. In a manual audit on Flickr30k, its flagged errors were correct 41% of the time versus 32.5% for CLIP similarity. We keep LEMoN's central insight but remove every cross-modal distance. LEMoN therefore remains exposed to the modality gap and to the circularity of using CLIP; our method is exposed to neither.

Learnable Expansion of Graph Operators (EGO) avoids the cross-modal feature gap by turning each modality into its own similarity graph and fusing graph powers. It reaches 97.26% AUC on ShanghaiTech with 0.091M parameters and loses less than 2% AUC under 50% feature noise. Its evidence that relational graphs are robust to feature noise supports our choice, and we reuse its multi-hop graph-power idea without learned weights. Unlike EGO, we work on the whole dataset rather than 32-sample mini-batches, and our task is data curation rather than anomaly classification.

Learning Robust Visual-Semantic Embeddings argues that pairwise contrastive losses pull matched points together while leaving the two distributions' overall shapes disjoint. That is one reason cross-modal cosine scores are an unreliable measure of quality.

Two lines of work constrain the design. SIEVE and BLIP's CapFilt show that caption-based filtering works: SIEVE plus CLIPScore beats CLIPScore alone by 2.6% on DataComp-Medium. That bar is the one to meet, and captioning cost is the reason to try another route. SIEVE also reports that text-similarity pruning fails on OCR and region-specific content, and our text-neighborhood signal may share that weakness. PRISM shows that anisotropy distorts similarity searches. Anisotropy here means embeddings bunching into a narrow cone. PRISM corrects it by re-centering, and we apply the same re-centering before building neighbor graphs.

Two methods supply machinery. CSA aligns frozen unimodal encoders (DINOv2, GTR) with a closed-form solution computed on a CPU and is reported to be robust to shuffled pairs. It is both a downstream consumer of our filtered data and, as discussed below, a serious competitor. The decoupled-inference literature supplies the systems pattern of heavy structure computed offline and cheap lookups online. Examples are GREED, which uses pair-independent embeddings with metric-tree indexing, and BRIGHT, which uses batch-computed graph embeddings served by lookup.

The Score

Let $N^{v}_k(i)$ be the $k$ nearest neighbors of pair $i$ under image similarity, and $N^{t}_k(i)$ the $k$ nearest under text similarity, both over the same set of pair indices. Let $A^v$ and $A^t$ be the corresponding row-normalized neighbor graphs. The agreement score is

$$s_i = \sum_{h=1}^{H} w_h \, \frac{\langle (A^v)^h_{i\cdot},\, (A^t)^h_{i\cdot} \rangle - \mu_i^{(h)}}{\sigma_i^{(h)}},$$

where $(A)^h_{i\cdot}$ is row $i$ of the $h$-step graph power, which spreads each neighborhood over $h$ hops. The inner product measures how much probability mass the two walks share. The quantities $\mu_i^{(h)}$ and $\sigma_i^{(h)}$ are the mean and spread of that overlap when text neighborhoods are randomly permuted among pairs with similar local text density. The weights $w_h$ are fixed and decay with $h$.

The normalization is the part that matters for rare concepts. A correct but rare pair has thin neighborhoods in both spaces, so its raw overlap is small. What distinguishes it from noise is that its overlap is well above what chance would give at that density. The one-hop version is simply the Jaccard-like overlap of the two neighbor sets.

Research Program

The build has three parts. The first is offline indexing. Embed every pair with a frozen image encoder (DINOv2) and a frozen text encoder (GTR or MiniLM), after SIEVE-style masking of boilerplate phrases such as "a photo of." Re-center each space as in PRISM. Build approximate nearest-neighbor indices for each.

The second is scoring. Each pair's score needs only its own index lookups plus sparse two-hop expansion. Scoring is therefore independent across pairs, parallel across shards, and applicable to new data without rebuilding anything.

The third is analysis. We will study which kinds of noise the score catches and which it misses, including generic alt-text, wrong-entity captions, OCR-dependent captions, and templated stock-photo text. We will also measure how much rare, correct content each filter discards, using held-out long-tail concept labels.

Baselines

The naive reference point is training on the unfiltered pool at the same compute, together with a random subset of the same size as each filtered set. This establishes how much any filtering helps at a given data budget.

The primary competitor is CLIPScore filtering, the standard DataComp baseline. We run it at the same kept fraction (sweeping 10–40%) and with the same downstream CLIP training recipe and compute. Beating it means a higher 38-task DataComp average at the best kept fraction for each method.

The strongest published filter is SIEVE combined with CLIPScore. On DataComp it is reported to beat CLIPScore alone by 2.6% on Medium and 1.7% on Large across 38 tasks. No absolute average appears in our file; we will rerun both on our infrastructure rather than import numbers. SIEVE is the strongest competitor because it also escapes CLIP circularity, but by paying for captioning. We will report its filtering cost in GPU-hours alongside ours. Our method wins if it matches SIEVE's gain at a small fraction of the filtering cost, or exceeds it when combined with CLIPScore.

For label-error detection we compare against LEMoN, using its protocol on CIFAR-10/100, Flickr30k, MSCOCO, MMIMDB and MIMIC-CXR. LEMoN is reported to beat 12 baselines by over 3% AUROC on classification and by up to 6.3% on captioning; our file gives no absolute per-dataset AUROC. Winning means higher AUROC on the same injected noise, plus a higher correct-flag rate in a 200-item manual audit on Flickr30k, where LEMoN's reported rate is 41%.

We also run a CSA-score baseline. We fit CSA on the full noisy pool from the same frozen encoders and score each pair by its aligned image-text similarity. This isolates exactly our key ingredient. Both methods use identical encoders and no image-text model; they differ only in whether the score comes from a cross-modal distance or from neighborhood agreement.

The ablations remove the density normalization, restrict to one hop ($H=1$), drop re-centering, and swap the encoders. They also include a LEMoN variant computed in CSA space, which tests whether our within-modality graphs beat neighborhood consensus inside a cheaply aligned shared space.

Why not just…?

Why not just use CLIPScore? It is cheap and it works; DataComp built DataComp-1B with CLIP-based filtering, and that data reached 79.2% ImageNet zero-shot with ViT-L/14. The objection to it is not that it fails. It measures one proxy model's opinion, so what it misses overlaps with what the trained model misses. SIEVE's gain from adding an independent signal is evidence of this. Our test is the same: if agreement scores add nothing on top of CLIPScore, the proposal has failed.

Why not just fit CSA on the noisy data and filter by its similarity? This is the real threat. CSA reports robustness to shuffled pairs and uses the same frozen unimodal encoders, so a CSA-score filter is equally free of CLIP and equally cheap. Our case against it is that CSA still produces a linear shared space, so it still depends on a direct cross-modal distance. It is also fit on the very noise it is supposed to find, and it is reported to lag on fine-grained large-scale retrieval. Whether this matters in practice is an empirical question, settled by the head-to-head CSA-score ablation. If CSA-score matches us on DataComp-Medium, the relational comparison is unnecessary and we will say so.

Why not just run LEMoN with a cheaper encoder? LEMoN needs a shared space to compute its image-to-text and text-to-image neighborhood distances. Swapping in unimodal encoders is impossible without first aligning them, for example with CSA, which brings back the cross-modal distance. We include exactly this LEMoN-in-CSA-space variant. If it wins, the lesson is that neighborhoods help but within-modality isolation does not, and we will report that.

Proposed Experiments and Decisive Checks

The central experiment uses the DataComp-Medium pool of 128M pairs with the fixed DataComp training recipe. Every filter above is swept over kept fractions, and we report the 38-task average, ImageNet zero-shot accuracy, and COCO/Flickr30k retrieval. Filtering cost is reported in GPU-hours and CPU-hours. The winning configuration is then checked at DataComp-Large (1.28B) against CLIPScore and SIEVE+CLIPScore.

Two secondary uses test whether the filtered data is useful beyond CLIP training. First, fit CSA on subsets of 35k to 1M pairs drawn from our filter versus random selection, and measure ImageNet zero-shot and Flickr30k retrieval. This asks whether clean selection extends CSA's data efficiency. Second, apply the score to CC3M at 5% and 10% budgets, comparing against ClipCov's reported 11.33% ImageNet zero-shot at 10%. Unlike ClipCov, we use no ImageNet class labels.

Three results are decisive.

First, at DataComp-Medium, agreement alone should at least match CLIPScore, and agreement plus CLIPScore should approach SIEVE's reported margin at under a tenth of SIEVE's filtering cost. If agreement alone falls clearly below CLIPScore and adds nothing in combination, the idea fails.

Second, agreement must beat CSA-score at identical encoders. If it does not, the within-modality comparison is not the source of any gain.

Third, the long-tail audit. The density-normalized score should discard correct rare-concept pairs at no higher a rate than CLIPScore. If it discards many more, the method has fixed noise by deleting the tail, and we would report that as a limitation rather than tune around it.