TL;DR
This paper introduces DataJuicer, a novel "juicing" approach to data governance that goes beyond simply selecting good image-text pairs. By fine-grainedly processing each sample to remove redundant image patches and enhance text descriptions with visual semantics, DataJuicer creates cleaner, more aligned datasets that significantly boost the performance of vision-language models.
What Problem Does This Paper Address?
Training large vision-language models (VLMs) requires massive datasets. However, simply scaling up datasets through unselective data collection leads to diminishing returns due to redundant and noisy samples. Existing methods, like "DataSieve," try to address this by selecting high-value image-text pairs. The problem is that these methods often operate at a coarse, sample-level and rely on heuristics, which can introduce biases and fail to remove inherent "undesired tokens" (e.g., irrelevant image patches or noisy text) within the retained samples. This limits the potential for further compression and purification of the data, ultimately impacting VLM performance.
Methods
DataJuicer introduces a "juicing" approach, moving beyond the "sieving" of DataSieve. It employs a dual-branch architecture:
- Vision Branch: This branch uses pre-trained vision foundation models (like DINO) to:
- Patchify Images: Divide images into smaller patches.
- Token-wise Reduction: Identify and retain only the most informative image patches based on their contribution to the overall image semantics, effectively removing redundant visual information.
- Object Class Extraction: Predict high-confidence object classes present in the image.
- Text Branch: This branch uses pre-trained language foundation models (like LLaMA) to:
- Enhance Captions: Incorporate the extracted object classes from the vision branch to enrich and refine the original text captions, improving image-text alignment.
- Denoise Text: Correct grammatical errors, misspellings, and repetitive expressions in the captions.
The combined output is a dataset with reduced redundancy, improved image-text consistency, and enhanced semantic clarity.
Datasets
The paper uses the following datasets:
- Training Datasets: CC3M, CC12M, YFCC15M, and a sampled subset of LAION40M.
- Testing Datasets: MSCOCO and Flickr30 (for image-text retrieval), ImageNet-1K (for image classification), and various benchmarks for multimodal large language model (MLLM) tasks like OKVQA and CCBench.
No new benchmarks were introduced; instead, the focus was on improving the quality of existing large-scale datasets.
Key Insights and Ideas
- "Juicing" vs. "Sieving": The core idea is that simply selecting good samples isn't enough. True data governance requires a finer-grained approach that actively "juices" the information out of each sample, removing noise and enhancing relevant signals.
- Token-Level Governance: Operating at the token (image patch or text word) level allows for much more precise control over data quality than sample-level selection.
- Synergy of Vision and Language Models: DataJuicer effectively leverages the strengths of both vision and language foundation models, using vision models to understand image content and language models to refine textual descriptions based on that understanding.
- Addressing Inherent Noise: The paper highlights that even "good" samples from web-scraped data can contain significant noise (e.g., irrelevant image regions, poorly written captions). DataJuicer directly tackles this.
- Complementarity with Existing Methods: DataJuicer can be used in conjunction with methods like DataSieve, further improving results.
Key Results
- Performance Gains: DataJuicer significantly outperforms existing methods like DataSieve across various downstream tasks, including image-text retrieval, classification, and dense visual reasoning. For example, on ImageNet-1K, DataJuicer achieved a 15.54% Top-1 accuracy with a 50% image retention ratio, compared to 11.99% for a "Random" baseline and 15.54% for DataSieve with a similar setup.
- Efficiency: DataJuicer offers faster inference speeds (e.g., 1.7x faster with a 50% image retention ratio) and comparable or lower training overhead compared to other methods.
- Scalability: DataJuicer demonstrates superior performance gains as data scale, model size, and training schedule increase, indicating its strong potential for training large foundation models.
- Generalization: The method shows robust generalization across different datasets (clean and noisy) and various vision-language architectures (CLIP, SigLIP, CoCa).
Limitations and Future Work
- Tokenization Dependency: DataJuicer's token-view approach requires models to embed data as tokens. While compatible with current architectures, it might not directly benefit novel architectures that deviate from this paradigm.
- Future Work: The paper suggests that exploring DataJuicer's applicability to other modalities and novel architectures would be a valuable direction. The authors also note that full-patch tuning (using all image patches during late training) can further boost performance on fine-grained tasks, albeit with a slight computational cost, suggesting this as an optional enhancement.
Summary updated