Vision-language models such as CLIP learn from billions of web images paired with their "alt-text," the short descriptions that page authors attach to images. Most of that text is junk, so pipelines prune it with a filter that asks whether the text matches the image. Our suspicion is that these filters quietly delete the rarest and most valuable words on the web. Suppose a photo is captioned "Pileated woodpecker, Dryocopus pileatus." A filter that scores image-text agreement with a general-purpose model sees a bird but does not recognize the species name, so it scores the pair as poorly aligned and throws it away. The pruned dataset then contains fewer rare terms, and the trained model is worse at exactly the long-tail recognition that filtering was supposed to protect.
We propose to borrow the oldest trick in lexical coverage expansion: look rare words up in a dictionary. Before a pair is scored, each rare term in its alt-text is temporarily replaced by its dictionary gloss or taxonomic parent ("woodpecker, a bird"). The filter judges whether the image is consistent with that generalized text. Pairs that pass keep their original, specific alt-text for training. In two sentences, the filter should decide whether a caption is grounded by checking a version of the caption it can understand, while the model still trains on the version that carries the rare word. The added cost is a dictionary lookup, against the weeks of accelerator time that recaptioning needs.
Core Claim
The claim has three parts. Pointwise filters systematically under-score pairs whose alt-text contains correct but rare terminology. That bias can be measured as a "lexical survival rate": how many lexicon terms still have enough training pairs after filtering. And replacing rare terms with dictionary glosses only at scoring time recovers much of the lost long-tail coverage without letting junk back in.
The evidence that motivates the gap comes from the noise side. Improving multimodal datasets with image captioning mixes synthetic captions into DataComp pools. Its gains shrink from +2% ImageNet at 128M samples to +1.2% at 1.28B, which the authors attribute to captioners' limited vocabulary. What If We Recaption Billions of Web Images with LLaMA-3? recaptions DataComp-1B and finds that the captioner cannot produce species names or proper nouns that were absent from the alt-text, so it falls back on an 80/20 web/synthetic mix.
The bias we target follows directly from how Sieve works. Sieve compares a synthetic caption with the alt-text using a sentence encoder. If the captioner says "a bird on a branch" and the alt-text names the species, the similarity is low precisely because the alt-text is more informative.
The method itself comes from lexical coverage work. Expanding Pretrained Models to Thousands More Languages via Lexicon-based Adaptation substitutes dictionary entries into text so that a model can place unknown words inside contexts it already understands. That paper uses substitution to create training data. We use it only to make a judgment fair, and the substituted text is never trained on.
The closest work on the noise side is VeCLIP, which uses a language model to fuse alt-text entities into visually enriched captions, and ALIP, which down-weights pairs according to how well raw text and synthetic caption agree. Both act on pairs that survive into training. Neither consults an external lexicon, and neither addresses the case where a rare-term pair is removed upstream by a filter. Generalizing Image Captions for Image-Text Parallel Corpus also generalizes captions. It does so to produce the training text itself, so its corpus loses the specific terms we want to keep.
Research Program
The pipeline has three steps.
- Term detection. A term counts as rare if it appears in an external lexicon and falls below a frequency threshold in the candidate pool. The main lexicon is a taxonomic tree of the kind BioTrove uses for its taxonomic prompts, mapping each species to its genus, family and vernacular name. For alt-text that is not in English, a PanLex-style bilingual lexicon supplies English glosses. This second use matters because an English sentence encoder will score a correct Portuguese caption as unaligned for the same underlying reason.
- Glossed scoring. Each rare term is replaced by its gloss, and the pair is scored with the unchanged filter. We test both CLIPScore and Sieve so the intervention stays separate from the choice of filter.
- Selection. Selection uses the glossed score, but the retained pair keeps its original alt-text.
Two known weaknesses of dictionary substitution constrain the design. The lexicon-adaptation paper documents polysemy errors such as "will" the verb versus "will" the testament. We therefore score every candidate gloss and record how often the choice of gloss changes the keep/drop decision. Huawei AARC's WMT21 biomedical submission found that short dictionary-like pairs without general data cause attention problems. Because we never train on glosses or dictionary entries, that failure mode does not arise. This is a deliberate difference from the injection approach.
The analysis component instruments every filter we study with the lexical survival rate: the fraction of lexicon terms that retain at least $k$ pairs after pruning, measured at several pool fractions. If glossing raises survival without raising measured noise, the claim holds.
Baselines
The do-nothing reference is the unfiltered DataComp pool and standard CLIPScore filtering, trained with the identical DataComp recipe.
The strongest filtering competitor that does not tune toward a benchmark is Sieve combined with CLIPScore. The file reports that this combination beats CLIPScore by 2.6% averaged over DataComp's 38 tasks at medium scale and by 1.7% at large scale. It reaches peak performance keeping 20% of the pool, against 30% for CLIPScore alone. The file gives no absolute 38-task average for Sieve, so we will rerun it ourselves. Our variant counts as beating it if glossed Sieve matches Sieve's 38-task average within noise across three seeds and also wins clearly on long-tail evaluations and survival rate. A long-tail gain bought with an average-task loss would not count.
We also report the target-tuned filters FLYT and SDM. They report 40.1% and 35.2% zero-shot ImageNet respectively on DataComp-medium. Both optimize toward ImageNet, so they serve as a reference ceiling on ImageNet. On concepts outside ImageNet we expect them to keep fewer rare terms, and we will check this with the survival metric.
The recaptioning competitor is the 80/20 web/synthetic mix from Recap-DataComp-1B. It is applied to the same filtered medium pool, so the comparison is about data and not about access to a captioner.
Two ablations isolate the dictionary ingredient. The first, the masking ablation, deletes rare terms before scoring instead of glossing them, much as Sieve already masks boilerplate. If masking does as well as glossing, the lexicon adds nothing. The second, the whitelist ablation, keeps every pair that contains a lexicon term regardless of score.
All runs hold fixed the pool, the number of retained samples, the architecture, the training steps and the threshold-tuning budget. Each threshold is chosen on a held-out slice that excludes the evaluation sets.
Why not just…?
Why not just lower the filter threshold so more rare pairs survive? Lowering the threshold admits rare-term pairs and junk at the same rate. Glossing is meant to admit only the former. The comparison at matched retained-pool size settles this directly. If a looser CLIPScore threshold reaches the same survival rate and the same 38-task average, our method is unnecessary. Improving multimodal datasets with image captioning shows that the unfiltered bottom of the pool is mostly unusable as raw text, which suggests the looser threshold will pay in average accuracy.
Why not just whitelist any pair containing a dictionary term? This is the real threat. It is cheaper still and needs no image check. Its weakness is that rare terms appear constantly in ungrounded contexts: product listings, file names, t-shirt shops naming a species they do not depict. A whitelist has no way to reject those. We run it as an ablation and audit 2,000 retained rare-term pairs by hand for whether the named entity is actually depicted. If the whitelist's grounding precision and downstream long-tail accuracy match glossed scoring, we will report that the image check is redundant and the contribution reduces to the survival diagnostic.
Why not just use a larger captioner or VeCLIP-style fusion? What If We Recaption Billions of Web Images with LLaMA-3? reports that an 8B-language-backbone captioner still misses species and proper nouns, and it needed 29 days on 256 TPU-v4 chips. VeCLIP can only carry forward entities that are still in the data after filtering. Fusion and glossing are therefore complementary. We test whether VeCLIP-style fusion applied after glossed filtering beats fusion applied after standard filtering.
Proposed Experiments
Development runs use DataComp-small (12.8M) for sweeps over the frequency threshold and the gloss depth (species, genus or family). The main runs use DataComp-medium (128M) with three seeds, reporting the 38-task average, ImageNet zero-shot, and the fine-grained classification tasks within the suite. The long-tail evaluation is zero-shot species identification on BioTrove-Balanced and BioTrove-Unseen. The file reports 91.1% top-1 on BioTrove-Balanced for BioTrove-CLIP, which was trained on curated expert data. We treat that number as a distant upper reference and do not expect web-trained models to approach it. Here the comparison that matters is between filters. The multilingual arm measures survival and retrieval for non-English alt-text glossed through the lexicon.
Three checks would be decisive.
- The bias is real. Standard filters should show measurably lower survival for lexicon terms than for frequency-matched common nouns. If they do not, the premise fails.
- Glossing helps where it should. Glossed scoring should improve BioTrove zero-shot accuracy at an equal retained fraction while keeping the 38-task average within seed variance of Sieve.
- The lexicon is the cause. Glossing should beat both the masking and whitelist ablations on the combination of grounding precision and long-tail accuracy.
As a final check, we will run one large-scale (1.28B) trial if the medium results hold. It tests whether glossing slows the shrinking returns that Improving multimodal datasets with image captioning observed as scale grows.
References
- Improving multimodal datasets with image captioning
- What If We Recaption Billions of Web Images with LLaMA-3?
- SIEVE: Multimodal Dataset Pruning Using Image Captioning Models
- Expanding Pretrained Models to Thousands More Languages via Lexicon-based Adaptation
- VeCLIP: Improving CLIP Training via Visual-Enriched Captions
- ALIP: Adaptive Language-Image Pre-training with Synthetic Caption
- Generalizing Image Captions for Image-Text Parallel Corpus
- BioTrove: A Large Curated Image Dataset Enabling AI for Biodiversity
- Huawei AARC's Submissions to the WMT21 Biomedical Translation Task: Domain Adaption from a Practical Perspective
- Filter Like You Test: Data-Driven Data Filtering for CLIP Pretraining
- Matched Data, Better Models: Target Aligned Data Filtering with Sparse Autoencoders