Introduction
Multimodal Large Language Models (MLLMs) have made significant strides in bridging the gap between vision and language. However, a primary bottleneck remains: the scarcity of high-quality, large-scale visual instruction tuning data. While models like LLaVA rely on datasets of approximately 158K samples, researchers from BAAI and Peking University have introduced SVIT, a massive dataset comprising 4.2 million visual instruction tuning pairs—a 20-fold increase in scale.
Scaling Visual Instruction Data
The core contribution of SVIT is the automated generation of high-quality instruction samples. Rather than manual labeling, the authors leveraged GPT-4 as a "teacher" model. By providing GPT-4 with rich human-annotated metadata from the Visual Genome dataset—including captions, region descriptions, and bounding boxes—the model could generate complex reasoning tasks without ever "seeing" the raw pixels.
The dataset is structured into four distinct task categories to ensure comprehensive multimodal capability:
- Conversation (1.6M): Multi-turn dialogues exploring the scene.
- Complex Reasoning (1.6M): "Why" and "How" questions that demand logical inference.
- Referring QAs (1.0M): Questions focused on specific image regions defined by spatial coordinates.
- Detailed Descriptions (106K): Exhaustive long-form narratives of the visual content.
The Data Recipe: Diversity and Balance
Scaling alone is insufficient; data quality and distribution are critical. The authors proposed a Coreset Selection Algorithm to refine the dataset. This strategy focuses on "Diversity"—selecting samples that span a wide array of visual concepts—and "Balance"—adjusting the ratio of "Yes/No" answers to mitigate the common "Yes-bias" found in GPT-generated data. Ablation studies confirmed that a diverse subset of 80K samples could outperform a randomly selected subset of the same size, highlighting the importance of data curation.
Experimental Results
The resulting model, SVIT-v1.5, was evaluated across 11 popular benchmarks, including MME, MMBench, and MMMU. SVIT-v1.5 (Full) consistently outperformed LLaVA-v1.5. Notably, on the MME benchmark, SVIT achieved a perception score of 1565.8, surpassing LLaVA-v1.5’s 1531.3. Even when utilizing efficient Low-Rank Adaptation (LoRA), SVIT-v1.5 outperformed the full-parameter version of LLaVA-v1.5 in cognitive reasoning tasks. The inclusion of 1.0 million referring QAs specifically bolstered the model's spatial accuracy and object counting abilities.
Conclusion and Future Work
SVIT demonstrates that scaling visual instruction data significantly enhances both the perception and cognition of MLLMs. While the dataset is subject to the reasoning limits of its teacher (GPT-4) and occasional errors in source metadata, it sets a new standard for open-source multimodal training. Future research may explore scaling further with video or 3D data to continue pushing the boundaries of artificial intelligence.
Summary updated