Denoising Web Scraped Datasets for Vision Language Models

725 concepts568 papers19 proposals

When training large-scale vision-language models (VLMs), the standard recipe relies on a simple premise: scrape billions of image-text pairs from the internet and optimize a contrastive objective like CLIP to map them into a shared latent space.

However, web-scraped data is fundamentally messy. Internet "alt-text" and social media captions are rarely literal descriptions of visual content. Instead, they are filled with metaphors, non-visual context, and outright misalignments. As observed by Polina Kuznetsova et al. in Generalizing Image Captions for Image-Text Parallel Corpus, human-written captions frequently reference entities, background stories, or circumstantial information that cannot be visually verified within the image frame.

When models are trained naively on this uncurated data, standard contrastive objectives break down. Understanding why this happens—and how modern architectures bypass this bottleneck—reveals a shift from scaling raw data quantity to engineering data quality.

The Geometry of Misalignment

To understand why web noise degrades performance, consider how contrastive learning operates. The objective forces the embedding of an image and its paired text to be identical (or highly correlated) while pushing all other unpaired images and texts in the batch apart.

Standard Contrastive Learning (Hard Targets):
[Image: Golden Retriever] <---> [Alt-text: "My favorite Sunday afternoon"] (Forced Alignment)

When the alt-text is completely unrelated to the literal visual features—such as an image of a dog captioned "My favorite Sunday afternoon"—the contrastive loss still forces the vision encoder and text encoder to pull these representations together. To minimize loss, the model must find spurious correlations, mapping unrelated visual features to irrelevant text tokens. As Enrico Fini et al. demonstrate in Improved baselines for vision-language pre-training, this uncurated noise forces standard training objectives to produce overconfident and poorly calibrated predictions, as they lack any inherent mechanism to account for data uncertainty.

To mitigate this, researchers have developed three primary paradigms: generative bootstrapping, soft-target distillation, and synthetic composition.

Strategy 1: Generative Bootstrapping and Filtering

If web-scraped alt-text is too noisy, one intuitive solution is to replace or filter it using a model trained on clean data. This is the core intuition behind the Captioning and Filtering (CapFilt) framework introduced by Junnan Li et al. in BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation.

The CapFilt Pipeline:

1. Web Image ---> [ Captioner ] ---> Synthetic Caption ("A golden retriever on grass")
2. [ Filter ] evaluates: Does the synthetic caption or original alt-text match the image better?
3. Cleaned pair is fed to the main VLM.

CapFilt uses two components: a captioner to generate synthetic web captions and a filter to remove noisy text. By replacing chaotic alt-text with literal, model-generated descriptions, the framework creates a semantic bottleneck. It strips away non-visual fluff and provides a clean, predictable learning objective.

Building on this concept, Anas Mahmoud et al. introduced Sieve: Multimodal Dataset Pruning Using Image Captioning Models. Instead of relying purely on a model's internal confidence scores (like CLIPScore), Sieve uses a captioning model to generate a synthetic description of the image and then uses a text-based sentence transformer to measure the semantic distance between the synthetic caption and the original web alt-text. This text-to-text comparison isolates and prunes pairs where the web text diverges too far from literal visual reality.

Strategy 2: Soft Targets via Momentum Distillation

An alternative to filtering out noisy data is changing the optimization objective to tolerate it. In a massive web dataset, a text caption might actually match multiple images in a training batch, or an image might be paired with a caption that is only partially true. Forcing a strict, binary one-hot target (where the pair must match perfectly and all other batch elements must be zero) introduces destructive gradients.

To solve this, Junnan Li et al. proposed ALBEF (Align before Fuse), which introduces Momentum Distillation.

Momentum Distillation Intuition:
Instead of: Target = [1.0 (this pair), 0.0, 0.0, 0.0 (all others)]
ALBEF uses: Target = [0.7 (this pair), 0.2 (similar image), 0.0, 0.1]

ALBEF maintains a momentum version of the vision and text encoders (an exponentially moving average of the training weights). Because the momentum model changes slowly, it acts as a stable teacher, generating continuous pseudo-targets for the training batch. If an image contains a dog and another image in the same batch also contains a dog, the momentum model will assign a soft, non-zero probability to that pseudo-pair. The training objective then optimizes against these soft targets rather than rigid, binary labels, preventing the model from penalizing correct associations that happen to be unlabelled in the dataset.

Similarly, Haoran Zhang et al. developed LEMoN: Label Error Detection using Multimodal Neighbors, which looks at the geometric consensus of neighborhoods in the joint latent space. If an image's nearest visual neighbors are all paired with text that is semantically distant from the target image's text, the pair is flagged as discordant. This leverages the collective geometry of the dataset to identify anomalies without relying on a single model's brittle zero-shot predictions.

Strategy 3: Bypassing the Web via Controlled Synthesis

As generative models have matured, a third paradigm has emerged: abandoning web scraping altogether in favor of fully synthetic data pipelines.

Web-scraped data often lacks descriptive depth, failing to capture complex spatial relationships or long-tail attributes. To bypass this, Qingqing Cao et al. developed CtrlSynth: Controllable Image Text Synthesis for Data-Efficient Multimodal Learning. Instead of collecting images and trying to fix their captions, CtrlSynth decomposes text into structured visual tags (objects, attributes, and relations) using a Large Language Model, and then uses a Diffusion model to generate corresponding images from scratch.

The CtrlSynth Pipeline:
[Structured Tags] ---> [LLM & Diffusion Model] ---> Perfectly Aligned Synthetic Pairs

This approach ensures perfect semantic-visual alignment by construction. Because the text and image are generated from the same underlying structured representation, there is no ambient noise, resulting in massive data efficiency gains during pretraining.

Quality Over Quantity

The collective evidence across these methodologies points to a clear conclusion: the sheer volume of web data no longer compensates for its lack of precision. Matt Deitke et al. demonstrated this shift explicitly with the release of Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Models. By prioritizing highly detailed, human-annotated data over billions of noisy web-scraped pairs, their models outperform significantly larger architectures trained on uncurated datasets.

Whether through generative filtering, soft-target optimization, or synthetic generation, managing the semantic gap between text and pixels is no longer just a data-cleaning step—it is a core architectural requirement for modern multimodal AI.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_web_scale_data_noise_ca839973,
  title = {web-scale data noise},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/web-scale-data-noise/rdr_ca839973d01243a68e8e08c0fd88ac37},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_web_scale_data_noise_ca839973_v1,
  title = {web-scale data noise},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/web-scale-data-noise/rdr_ca839973d01243a68e8e08c0fd88ac37/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake