Quality-Centric Alignment for Vision-Language Models

This direction is merged into another direction: web-scale data noise. Its last version stays readable and citable here.

Version 1 of 1303 papers317 conceptsfrozen 2026-10-04

The prevailing wisdom in multimodal learning has long been that "scale is all you need." By scraping billions of image-text pairs from the web, models like CLIP learned to associate visual concepts with natural language. However, this web-scale data is notoriously "noisy": alt-text is often irrelevant, captions are frequently ungrammatical hashtags, and many images are aesthetically poor or misaligned with their descriptions.

Recent research has shifted focus from sheer volume to quality-centric alignment. This direction posits that the bottleneck in vision-language models (VLMs) is no longer the number of parameters or the size of the dataset, but the integrity of the correspondence between pixels and text. By filtering, rewriting, or synthetically generating high-quality training data, researchers are achieving state-of-the-art performance with a fraction of the original data.

The Cost of Web-Scale Noise

Web-scraped datasets contain a high percentage of "weakly corresponding" pairs. As noted in Wukong by Jiaxi Gu, these misalignments prevent models from learning precise visual-semantic mappings. When a model is trained on a caption that describes metadata (e.g., "IMG_4021.jpg") rather than the visual scene, it fails to ground linguistic concepts in specific image regions.

This noise leads to several failure modes:

  1. Hallucinations: Models trained on noisy data often "see" objects that aren't there because they've learned spurious correlations from inaccurate captions DeepSeek-VL2.
  2. Poor Compositional Reasoning: Brief or generic alt-text lacks the descriptive depth needed for a model to understand complex relationships between objects ImageInWords.
  3. Degraded Generative Quality: In text-to-image tasks, training on aesthetically poor or misaligned samples results in inconsistent generation Janus-Pro.

Filtering and Pruning the Long Tail

One primary strategy for improving alignment is dataset pruning. Instead of treating every web-scraped pair as ground truth, researchers use "teacher" models to identify and remove low-quality samples.

Anas Mahmoud and colleagues introduced Sieve, which uses small, high-quality captioning models to generate "synthetic" ground truth for web images. By comparing these synthetic captions to the original web text, Sieve can prune samples where the two disagree, significantly outperforming standard metrics like CLIPScore. Similarly, Haoran Zhang proposed LEMoN, which detects label errors by checking if an image's "multimodal neighbors" share similar semantic content. If an image is surrounded by "cat" images but is labeled "dog," it is flagged as noise.

The DataComp Challenge results from Shuhei Yokoo further validated this, showing that filtering a dataset down to its most relevant 10% can actually improve downstream performance compared to training on the full, noisy set.

Synthetic Refinement and Rewriting

If the original web text is poor, why not replace it? A growing body of work focuses on caption modification. Shuhei Yokoo demonstrated that using a model like BLIP-2 to rewrite web captions into clean, descriptive sentences provides a massive boost in zero-shot performance.

Taking this a step further, Dharshan Kumaran introduced Synth², a framework that trains VLMs using entirely synthetic data—LLM-generated captions paired with image embeddings from text-to-image models. This bypasses the noise of the web entirely, allowing for a perfectly aligned (albeit simulated) training environment.

Robust Training Objectives

When noise cannot be filtered out, the training objective itself must become robust to it. Junnan Li addressed this in ALBEF through Momentum Distillation. ALBEF maintains a "momentum" version of the model that generates pseudo-targets for the image-text pairs. If the web caption is noisy, the momentum model’s own internal representation acts as a more reliable target, effectively "smoothing" the noise during the alignment process.

In the generative space, Weichen Yu proposed dynamically down-weighting unreliable tokens during training. By using a VLM's confidence score to identify potential hallucinations in a caption, the model learns to ignore the "noisy" parts of the text while focusing on the visually grounded elements.

The Shift to High-Quality Human Curation

The most recent trend suggests a return to small-scale, high-quality human annotation. Matt Deitke and the team behind Molmo demonstrated that a 72B parameter model trained on a meticulously curated, human-annotated dataset can outperform models trained on billions of web samples.

This is echoed by Roopal Garg in ImageInWords, which uses a human-in-the-loop framework to produce "hyper-detailed" descriptions. These descriptions go beyond simple labels to describe spatial relationships, textures, and styles. The evidence suggests that for the next generation of VLMs, data quality—specifically the precision of the alignment between visual and textual concepts—is the most potent lever for performance.

Papers in this version

Showing 60 of 303 papers.

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_quality_centric_alignment_for_vision_lan_d5ac4349,
  title = {Quality-Centric Alignment for Vision-Language Models},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/quality-centric-alignment-for-vision-language-models/rdr_d5ac43498ccd46749812b6fb27224844},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_quality_centric_alignment_for_vision_lan_d5ac4349_v1,
  title = {Quality-Centric Alignment for Vision-Language Models},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/quality-centric-alignment-for-vision-language-models/rdr_d5ac43498ccd46749812b6fb27224844/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}