Abstract
Traditional Vision Language Model (VLM) training often relies on the brute-force scaling of parameters and uncurated datasets. We present Data Metabolism, a data-centric framework that treats training data as an evolving resource rather than a static input. By implementing a closed-loop system of data construction (Anabolism) and diagnostic refinement (Catabolism), we developed Capybara-VL-7B. Despite its modest parameter count, Capybara-VL-7B achieves state-of-the-art performance on multimodal benchmarks, frequently rivaling or surpassing models with over 70 billion parameters.
Introduction
The prevailing trend in VLM development emphasizes scaling parameters and dataset sizes. However, this approach often leads to diminishing returns and opaque failure modes where it is unclear if errors stem from architecture or "toxic" data. Researchers at Kuaishou Technology propose a shift toward data efficiency through "Data Metabolism." This framework moves beyond static data mixtures, introducing an iterative process to identify, prune, and improve training samples based on model performance.
The Data Metabolism Framework
The core of the proposed methodology is a two-phase cycle designed to optimize the "nutritional value" of training data through continuous feedback.
1. Data Anabolism (Construction)
This phase focuses on synthesizing high-quality training samples. It involves:
- Multi-stage Filtering: Utilizing perceptual hashing for deduplication and advanced models (e.g., Qwen2-VL, GPT-4o) for relevance and quality scoring.
- Answer Improvement: Transforming short labels into verbose narratives and generating Chain-of-Thought (CoT) reasoning paths to enhance the model's explanatory capabilities and reduce format sensitivity.
2. Data Catabolism (Diagnosis)
This phase involves "digesting" model failures to refine the dataset. The researchers categorize failures into specific buckets: Unexpected Behavior (refusals), Hallucination, Task-format Inconsistency, and Cross-modal Inconsistency (where text knowledge exists but visual perception fails). By identifying these gaps, the team can perform Incremental Validation and Redundant Data Pruning, removing simple data the model has already mastered to make room for complex reasoning tasks.
Experimental Results
The researchers "metabolized" a raw pool of 38 million samples down to a high-quality core of 12 million. Capybara-VL-7B was trained using a three-stage pipeline: Visual Concept Alignment, Detailed Image Understanding, and Instruction Tuning.
As shown in the results, Capybara-VL-7B demonstrates exceptional efficiency. On the MathVista benchmark, it achieved a score of 70.7, outperforming the 76B-parameter InternVL2 (65.5) and surpassing the 72B-parameter Qwen2-VL (70.5). Similar trends were observed in OCR tasks, where it scored 876 on OCRBench, surpassing nearly all open-source competitors regardless of size. The model achieves parity with proprietary models like GPT-4o-mini and Gemini 1.5 Pro on several reasoning benchmarks while remaining small enough to run on consumer hardware.
Conclusion
Data Metabolism proves that strategic data curation can compensate for smaller parameter counts. By treating data as a dynamic resource and focusing on diagnostic-driven refinement, Capybara-VL-7B achieves a new frontier in VLM efficiency. Future work will focus on injecting new visual knowledge during instruction tuning to overcome the limitations of frozen visual encoders.
Summary updated