Squeeze Out Tokens from Sample for Finer-Grained Data Governance

Weixiong Lin, Chen Ju, Haicheng Wang, Shengchao Hu, Shuai Xiao, Mengting Chen, Yuheng Jiao, Mingshuai Yao, Jinsong Lan, Qingwen Liu, Ying ChenarXiv 2025 · DOI · OpenReview

TL;DR

This paper introduces DataJuicer, a novel "juicing" approach to data governance that goes beyond simply selecting good image-text pairs. By fine-grainedly processing each sample to remove redundant image patches and enhance text descriptions with visual semantics, DataJuicer creates cleaner, more aligned datasets that significantly boost the performance of vision-language models.

What Problem Does This Paper Address?

Training large vision-language models (VLMs) requires massive datasets. However, simply scaling up datasets through unselective data collection leads to diminishing returns due to redundant and noisy samples. Existing methods, like "DataSieve," try to address this by selecting high-value image-text pairs. The problem is that these methods often operate at a coarse, sample-level and rely on heuristics, which can introduce biases and fail to remove inherent "undesired tokens" (e.g., irrelevant image patches or noisy text) within the retained samples. This limits the potential for further compression and purification of the data, ultimately impacting VLM performance.

Methods

DataJuicer introduces a "juicing" approach, moving beyond the "sieving" of DataSieve. It employs a dual-branch architecture:

  • Vision Branch: This branch uses pre-trained vision foundation models (like DINO) to:
  • Patchify Images: Divide images into smaller patches.
  • Token-wise Reduction: Identify and retain only the most informative image patches based on their contribution to the overall image semantics, effectively removing redundant visual information.
  • Object Class Extraction: Predict high-confidence object classes present in the image.
  • Text Branch: This branch uses pre-trained language foundation models (like LLaMA) to:
  • Enhance Captions: Incorporate the extracted object classes from the vision branch to enrich and refine the original text captions, improving image-text alignment.
  • Denoise Text: Correct grammatical errors, misspellings, and repetitive expressions in the captions.

The combined output is a dataset with reduced redundancy, improved image-text consistency, and enhanced semantic clarity.

Datasets

The paper uses the following datasets:

  • Training Datasets: CC3M, CC12M, YFCC15M, and a sampled subset of LAION40M.
  • Testing Datasets: MSCOCO and Flickr30 (for image-text retrieval), ImageNet-1K (for image classification), and various benchmarks for multimodal large language model (MLLM) tasks like OKVQA and CCBench.

No new benchmarks were introduced; instead, the focus was on improving the quality of existing large-scale datasets.

Key Insights and Ideas

  • "Juicing" vs. "Sieving": The core idea is that simply selecting good samples isn't enough. True data governance requires a finer-grained approach that actively "juices" the information out of each sample, removing noise and enhancing relevant signals.
  • Token-Level Governance: Operating at the token (image patch or text word) level allows for much more precise control over data quality than sample-level selection.
  • Synergy of Vision and Language Models: DataJuicer effectively leverages the strengths of both vision and language foundation models, using vision models to understand image content and language models to refine textual descriptions based on that understanding.
  • Addressing Inherent Noise: The paper highlights that even "good" samples from web-scraped data can contain significant noise (e.g., irrelevant image regions, poorly written captions). DataJuicer directly tackles this.
  • Complementarity with Existing Methods: DataJuicer can be used in conjunction with methods like DataSieve, further improving results.

Key Results

  • Performance Gains: DataJuicer significantly outperforms existing methods like DataSieve across various downstream tasks, including image-text retrieval, classification, and dense visual reasoning. For example, on ImageNet-1K, DataJuicer achieved a 15.54% Top-1 accuracy with a 50% image retention ratio, compared to 11.99% for a "Random" baseline and 15.54% for DataSieve with a similar setup.
  • Efficiency: DataJuicer offers faster inference speeds (e.g., 1.7x faster with a 50% image retention ratio) and comparable or lower training overhead compared to other methods.
  • Scalability: DataJuicer demonstrates superior performance gains as data scale, model size, and training schedule increase, indicating its strong potential for training large foundation models.
  • Generalization: The method shows robust generalization across different datasets (clean and noisy) and various vision-language architectures (CLIP, SigLIP, CoCa).

Limitations and Future Work

  • Tokenization Dependency: DataJuicer's token-view approach requires models to embed data as tokens. While compatible with current architectures, it might not directly benefit novel architectures that deviate from this paradigm.
  • Future Work: The paper suggests that exploring DataJuicer's applicability to other modalities and novel architectures would be a valuable direction. The authors also note that full-patch tuning (using all image patches during late training) can further boost performance on fine-grained tasks, albeit with a slight computational cost, suggesting this as an optional enhancement.

Summary updated

Abstract

Widely observed data scaling laws, in which error falls off as a power of the training size, demonstrate the diminishing returns of unselective data expansion. Hence, data governance is proposed to downsize datasets through pruning non-informative samples. Yet, isolating the impact of a specific sample on overall model performance is challenging, due to the vast computation required for tryout all sample combinations. Current data governors circumvent this complexity by estimating sample contributions through heuristic-derived scalar scores, thereby discarding low-value ones. Despite thorough sample sieving, retained samples contain substantial undesired tokens intrinsically, underscoring the potential for further compression and purification. In this work, we upgrade data governance from a 'sieving' approach to a 'juicing' one. Instead of scanning for least-flawed samples, our dual-branch DataJuicer applies finer-grained intra-sample governance. It squeezes out informative tokens and boosts image-text alignments. Specifically, the vision branch retains salient image patches and extracts relevant object classes, while the text branch incorporates these classes to enhance captions. Consequently, DataJuicer yields more refined datasets through finer-grained governance. Extensive experiments across datasets demonstrate that DataJuicer significantly outperforms existing DataSieve in image-text retrieval, classification, and dense visual reasoning.
BibTeX
@misc{lin2025squeeze,
  author = {Weixiong Lin and Chen Ju and Haicheng Wang and Shengchao Hu and Shuai Xiao and Mengting Chen and Yuheng Jiao and Mingshuai Yao and Jinsong Lan and Qingwen Liu and Ying Chen},
  title = {{Squeeze Out Tokens from Sample for Finer-Grained Data Governance}},
  year = {2025},
  eprint = {2503.14559},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2503.14559},
}
Figure