Telling Wrong Captions from Short Ones in Web Image–Text Data

Web-scraped image–text collections are filtered by scoring how well each caption matches its image and discarding the lowest scores. That rule cannot distinguish two different failures. One is a caption that is simply wrong, such as a photo of a dog labelled "airplane". The other is a caption that is correct but nearly empty, such as a street-market photo labelled "Tokyo" or a holiday snapshot labelled "Sunday". Both get low scores, and both get thrown away. Discarding the second kind is costly. It removes much of the colloquial, non-English and non-commercial content in the crawl, which is how automated filtering ends up amplifying who is excluded from the data.

We propose to separate the two failures by asking a different question of short captions. Instead of encoding a two-word caption into a vector, which carries little information, we treat each short caption string as a label. The billions of images that the web attaches to that same string are its evidence. From the images paired with "Tokyo" across the pool, we can measure two separate things. The first is whether this image is typical of the images the web calls "Tokyo", which gives a calibrated alignment test. The second is how tightly the string constrains its images at all, which gives a specificity score. A wrong caption fails the first test. A short-but-honest caption passes the first test and scores low on the second. The filter then decides what to do with each case separately, rather than collapsing both into one low number.

Core Claim

For captions of roughly one to four tokens, grouping pairs by normalized caption string and testing each image against its group's image distribution will separate mismatched pairs from sparse-but-correct pairs better than any per-pair or embedding-neighborhood score. The alignment test is a conformal test: it compares the image against the group's own images and yields a p-value whose error rate is guaranteed when exchangeability holds. Filtering by this test should match or beat the best existing filters on DataComp downstream accuracy while retaining substantially more short, non-English and culturally specific pairs.

The idea combines the three input directions in fixed roles. Web-scale noise is the problem. Short-text sparsity is the regime where existing detectors break. Automated corpus construction supplies the mechanism: the crawl itself acts as an annotator. This follows the logic of An Automatic Method for Generating Sense Tagged Corpora, which harvested web contexts of unambiguous "monosemous relatives" to label ambiguous words automatically. Our strings with tight image groups play the role of visually monosemous labels. Strings with diffuse groups are the visually polysemous ones.

Why existing detectors miss this case

Three filters attack noise at the pair level, and each hits the short-caption wall in its own way. SIEVE captions the image with BLIP, masks boilerplate, and compares that synthetic caption to the alt-text with a small sentence encoder. A one-token alt-text gives that encoder almost nothing to compare. ADCluster documents the broader problem: sentence embeddings of truncated texts crowd into overlapping regions. The conformal CLIPScore work, A Conformal Risk Control Framework for Granular Word Assessment and Uncertainty Calibration of CLIPScore Quality Estimates, builds score distributions by masking image patches and caption words. The authors state this is "less effective for extremely short captions". Their work motivates our use of conformal calibration. We obtain the calibration distribution from other images sharing the string instead of from masking a caption that has nothing left to mask.

The closest work is LEMoN. It adds two neighborhood checks to pairwise CLIP distance. One asks whether the image's visual neighbors carry similar captions. The other asks whether the caption's text neighbors are paired with similar images. LEMoN improves captioning-noise detection by up to 6.3% AUROC over baselines. Our method differs in three ways. LEMoN finds text neighbors by embedding similarity, which is exactly what degrades for sparse text; we group by exact normalized string and back off to embeddings only when a string is rare. LEMoN outputs one uncalibrated discordance score; we output a conformal p-value plus a separate specificity score, so "wrong" and "vague" stay distinct. Finally, LEMoN was evaluated as an offline detector on curated captioning sets, not as a pool filter scored on DataComp or audited for exclusion.

Two papers constrain the design. DataComp shows that aggressive CLIPScore filtering is what makes strong subsets such as DataComp-1B. Who's in and who's out? shows the same operation widens demographic gaps, for example from 42.1% to 63.9% between male- and female-imputed faces. A useful filter must therefore stay competitive on accuracy while improving retention. Doing better on retention alone would not be enough.

Research Program

The pool is DataComp CommonPool, with the image and text embeddings DataComp already provides. Captions are normalized by lowercasing, stripping punctuation and Unicode-normalizing. Diacritic-insensitive matching is an optional variant, motivated by the diacritic loss documented in NaijaSenti. Captions of at most $L$ tokens are then hashed into groups. Grouping is a single pass with a hash table, and the per-group statistics are means and nearest-neighbor distances over precomputed vectors. The cost is therefore far below generative recaptioning, and the method runs on any lab's hardware.

For a pair with image embedding $v$ in a group $G$ of size $n$, the nonconformity score $s(v) = 1 - \cos(v, \mu_{G \setminus v})$ is one minus the cosine similarity between the image and the leave-one-out group mean. A robust alternative uses the median similarity to the $k$ nearest group members instead of the mean. The conformal p-value is

$$p(v) = \frac{1 + |\{u \in G \setminus v : s(u) \ge s(v)\}|}{n},$$

which counts the fraction of group members at least as atypical as $v$. If group members are exchangeable, then $P(p(v) \le \alpha) \le \alpha$ for correctly paired images. This bound is what lets us set a target false-rejection rate for honest pairs instead of a raw score threshold.

Exchangeability is the key assumption, and it can fail. A group may be a mixture of a true concept and a spam cluster in which one stock image is reposted under many strings. We therefore deduplicate near-identical images within each group before computing scores, and we report calibration empirically on held-out clean groups.

Specificity is defined as $\kappa(G) = \cos$-mean of group members to $\mu_G$. It is high when the string's images look alike, as with "golden retriever", and low when they do not, as with "Sunday".

A pair is rejected as misaligned when $p(v) < \alpha$. It is flagged as sparse when it passes but $\kappa(G)$ is low. Sparse pairs are kept but capped per string, so a diffuse string cannot dominate the batch. Strings with fewer than $n_{\min}$ members back off to a group of embedding neighbors in text space, which recovers a LEMoN-style test exactly where we have no better option. We will report what fraction of pairs fall into this backoff, because a large backoff share would mean the string-grouping idea does not reach the regime it targets. Captions longer than $L$ tokens are scored by the existing best filter, so the method only replaces the pairwise score where that score is known to be weakest.

Baselines

The naive reference point is no filtering at all, together with random subsampling at the same retained fraction. Both establish the floor on the DataComp small and medium filtering tracks.

The standard competitor is the DataComp CLIPScore baseline DataComp. It keeps the top fraction of pairs by image–text cosine similarity, with the fraction swept as in the benchmark. It is cheap, and our method must beat it, not merely match it, to justify any additional complexity.

The strongest existing filter in the file is SIEVE. It reports gains over CLIPScore of up to 2.6% averaged downstream on DataComp medium (128M) and 1.7% on large (1.28B). It also reaches peak performance with 20% of the data versus 30% for CLIPScore. The file gives these only as gains, and the absolute average DataComp score for SIEVE at medium scale is not in the file: no reported result found on Lacuna for it. We will therefore rerun SIEVE ourselves with its released settings on the same pool shard.

All filters will run on the identical candidate pool with the fixed DataComp training recipe and compute budget. Each method gets the same sweep of retained fractions, chosen on the same validation tasks. Beating SIEVE means a higher 38-task DataComp average at the best retained fraction, with a confidence interval from at least three training seeds. It also means no loss on ImageNet zero-shot accuracy.

LEMoN is run with the same CLIP embeddings and the same neighbor index. It isolates whether our gain comes from exact-string grouping or from neighborhood consensus in general.

Three ablations isolate the ingredients. Replacing string grouping with embedding neighbors everywhere tests the central claim. Dropping $\kappa$ and treating every passing pair alike tests whether separating "vague" from "fine" matters. Replacing the conformal threshold with a raw percentile cut tests whether calibration earns its place.

Why not just…?

Why not just drop every caption shorter than ten tokens? This is the cheapest version of the problem and must be run as a baseline. The evidence against it is indirect. The DataComp exclusion audit Who's in and who's out? shows that heuristic filtering disproportionately removes non-English and non-commercial text. Short captions plausibly carry exactly that content, and NaijaSenti shows that real social-media text in low-resource languages is short and heavily code-mixed, with 43% of Igbo tweets code-mixed. Whether the length filter actually costs downstream accuracy is an empirical question. If it matches our method on both DataComp average and the retention audit, the proposal fails, and we will say so.

Why not just recaption every short-caption image with BLIP or an LLM? This is the most serious threat. Replacing sparse text with dense synthetic text could beat any filter on accuracy. Three costs count against it. First, it discards the human string, and with it the non-English and culturally specific vocabulary we are trying to keep. Second, captioners omit OCR, geolocation and fine-grained detail, as SIEVE itself reports. Third, an LLM elaborating "Sunday" will invent non-visual context, the very noise that Generalizing Image Captions for Image-Text Parallel Corpus identifies in human captions. Cost also matters: GenCo Generation-driven Contrastive Self-training for Zero-shot Text Classification with Instruction-following LLM reports 9 minutes per 10k instances for a 7B model on an A6000. We will run captioner recaptioning of all short-caption pairs as a baseline at small scale. If it wins on accuracy, the right conclusion is a hybrid: our test decides which short pairs are worth recaptioning or keeping. We will test that hybrid directly instead of claiming the filter alone suffices.

Why not just use LEMoN as is? LEMoN already uses neighborhoods, and on many pools its text-neighbor term may approximate exact-string grouping. The embedding-neighbor ablation and the direct LEMoN baseline settle this. If our advantage does not concentrate on captions of four tokens or fewer, it is not real.

Proposed Experiments

The first experiment is a controlled diagnostic run before any pretraining. We take MSCOCO and Flickr30k pairs, using the captioning-noise setting from LEMoN, and create two corruptions. Sparse-but-correct pairs truncate a caption to its head noun or a one-to-three-token keyword phrase. Wrong pairs swap in a keyword from another image. Groups are formed over a large background pool so that strings recur. The metrics are AUROC for detecting wrong pairs, and the false-rejection rate on sparse-but-correct pairs at a fixed recall of wrong pairs. CLIPScore, SIEVE and LEMoN are compared, all stratified by caption length. We will also check whether the empirical false-rejection rate on clean truncated pairs tracks $\alpha$, which tests calibration. The decisive result is a large reduction in false rejection of sparse-but-correct pairs at equal wrong-pair recall. Without it, the downstream experiments are moot.

The second experiment is the DataComp filtering track at small (12.8M) and medium (128M) scale. We report the 38-task average, ImageNet zero-shot accuracy and retrieval against all baselines above. For context only, DataComp reports 79.2% ImageNet zero-shot for DataComp-1B at far larger compute, which is not comparable to these scales.

The third experiment is the exclusion audit, run on every filtered subset. It uses the methodology of Who's in and who's out?: the gap in imputed face gender, retention by detected language, and retention of non-English short strings. Language is identified with a standard identifier, and with AfroLID for African languages. Success means retaining clearly more non-English short-caption pairs than CLIPScore at matched downstream accuracy, without widening the demographic gap.

The fourth experiment is an inspection of the by-product. The ranked list of high-specificity short strings and their image groups is an automatically constructed visual lexicon of web vocabulary. We will hand-audit a sample per language to measure whether high-$\kappa$ strings are in fact visually coherent concepts. If they are not, the specificity score is not doing what we claim.