SVIT: Scaling up Visual Instruction Tuning

Bo Zhao, Boya Wu, Tiejun HuangarXiv 2023 · DOI · OpenReview

Introduction

Multimodal Large Language Models (MLLMs) have made significant strides in bridging the gap between vision and language. However, a primary bottleneck remains: the scarcity of high-quality, large-scale visual instruction tuning data. While models like LLaVA rely on datasets of approximately 158K samples, researchers from BAAI and Peking University have introduced SVIT, a massive dataset comprising 4.2 million visual instruction tuning pairs—a 20-fold increase in scale.

Scaling Visual Instruction Data

The core contribution of SVIT is the automated generation of high-quality instruction samples. Rather than manual labeling, the authors leveraged GPT-4 as a "teacher" model. By providing GPT-4 with rich human-annotated metadata from the Visual Genome dataset—including captions, region descriptions, and bounding boxes—the model could generate complex reasoning tasks without ever "seeing" the raw pixels.

SVIT model architecture and task categories including Captioning, Perception, Reasoning, and Referring.

The dataset is structured into four distinct task categories to ensure comprehensive multimodal capability:

  1. Conversation (1.6M): Multi-turn dialogues exploring the scene.
  2. Complex Reasoning (1.6M): "Why" and "How" questions that demand logical inference.
  3. Referring QAs (1.0M): Questions focused on specific image regions defined by spatial coordinates.
  4. Detailed Descriptions (106K): Exhaustive long-form narratives of the visual content.

The Data Recipe: Diversity and Balance

Scaling alone is insufficient; data quality and distribution are critical. The authors proposed a Coreset Selection Algorithm to refine the dataset. This strategy focuses on "Diversity"—selecting samples that span a wide array of visual concepts—and "Balance"—adjusting the ratio of "Yes/No" answers to mitigate the common "Yes-bias" found in GPT-generated data. Ablation studies confirmed that a diverse subset of 80K samples could outperform a randomly selected subset of the same size, highlighting the importance of data curation.

Experimental Results

The resulting model, SVIT-v1.5, was evaluated across 11 popular benchmarks, including MME, MMBench, and MMMU. SVIT-v1.5 (Full) consistently outperformed LLaVA-v1.5. Notably, on the MME benchmark, SVIT achieved a perception score of 1565.8, surpassing LLaVA-v1.5’s 1531.3. Even when utilizing efficient Low-Rank Adaptation (LoRA), SVIT-v1.5 outperformed the full-parameter version of LLaVA-v1.5 in cognitive reasoning tasks. The inclusion of 1.0 million referring QAs specifically bolstered the model's spatial accuracy and object counting abilities.

Conclusion and Future Work

SVIT demonstrates that scaling visual instruction data significantly enhances both the perception and cognition of MLLMs. While the dataset is subject to the reasoning limits of its teacher (GPT-4) and occasional errors in source metadata, it sets a new standard for open-source multimodal training. Future research may explore scaling further with video or 3D data to continue pushing the boundaries of artificial intelligence.

Summary updated

Abstract

Thanks to the emerging of foundation models, the large language and vision models are integrated to acquire the multimodal ability of visual captioning, question answering, etc. Although existing multimodal models present impressive performance of visual understanding and reasoning, their limits are still largely under-explored due to the scarcity of high-quality instruction tuning data. To push the limits of multimodal capability, we Scale up Visual Instruction Tuning (SVIT) by constructing a dataset of 4.2 million visual instruction tuning data including 1.6M conversation question-answer (QA) pairs, 1.6M complex reasoning QA pairs, 1.0M referring QA pairs and 106K detailed image descriptions. Besides the volume, the proposed dataset is also featured by the high quality and rich diversity, which is generated by prompting GPT-4 with the abundant manual annotations of images. We also propose a new data recipe to select subset with better diversity and balance, which evokes model's superior capabilities. Extensive experiments verify that SVIT-v1.5, trained on the proposed dataset, outperforms state-of-the-art Multimodal Large Language Models on popular benchmarks. The data and code are publicly available at https://github.com/BAAI-DCAI/Visual-Instruction-Tuning.
BibTeX
@misc{zhao2023svit,
  author = {Bo Zhao and Boya Wu and Tiejun Huang},
  title = {{SVIT: Scaling up Visual Instruction Tuning}},
  year = {2023},
  eprint = {2307.04087},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2307.04087},
}
Figure