ALLaVA: Harnessing GPT4V-synthesized Data for A Lite Vision-Language Model

Guiming Hardy Chen, Shunian Chen, Ruifei Zhang, Junying Chen, Xiangbo Wu, Zhiyi Zhang, Zhihong Chen, Jianquan Li, Xiang Wan, Benyou Wang

Introduction

Large Vision-Language Models (LVLMs) such as GPT-4V and Qwen-VL have set new benchmarks in multimodal understanding. However, their massive parameter counts make deployment on edge devices like smartphones impractical. While "lite" LVLMs exist, they often suffer from a significant performance gap compared to their larger counterparts. The ALLaVA project posits that this gap is primarily due to data quality rather than just model scale. Most small models are trained on noisy datasets like LAION, which feature short, coarse-grained captions that lack the fine-grained detail necessary for complex reasoning.

The ALLaVA Methodology: Captioning-then-QA

To address the data quality issue, the authors developed the ALLaVA dataset using a novel "Captioning-then-QA" pipeline. Instead of generating a simple question-answer pair, the teacher model (GPT-4V) follows a three-stage process:

  1. Detailed Captioning: Generating a fine-grained description of the image.
  2. Complex Questioning: Formulating a reasoning-based question grounded in the description.
  3. Detailed Answering: Providing a comprehensive answer.
The ALLaVA data generation pipeline using GPT-4V for LAION and Vision-FLAN sources.

This "thinking out loud" approach ensures that the generated instructions are visually grounded, significantly reducing hallucinations. Manual evaluation showed that this method achieves 84% accuracy, compared to 78% for direct QA generation, proving that forcing the model to describe the scene first improves the quality of subsequent reasoning.

Dataset Composition and Diversity

The resulting ALLaVA dataset comprises 1.3 million samples split into two main components:

  • ALLaVA-Caption-4V (664K): Used for initial vision-language alignment.
  • ALLaVA-Instruct-4V (663K): Used for visual instruction tuning.

The data is sourced from a curated subset of LAION-400M, covering over 122,000 unique web domains to ensure natural diversity, and Vision-FLAN, which provides a wide array of task-specific instructions across 101 open-source datasets.

Experimental Results and Scale Compensation

The researchers trained 4B-scale models (e.g., ALLaVA-Phi3) and evaluated them across 17 benchmarks. The results demonstrate that high-quality synthetic data acts as "scale compensation," allowing smaller models to rival or even outperform much larger counterparts trained on noisier data.

Performance comparison showing the significant gains achieved by Mipha-3B after training on the ALLaVA dataset.

Key findings include:

  • Superior Performance: ALLaVA-Phi3 outperformed LLaVA-1.5-7B on the LLaVA-Bench (In-the-Wild) with a score of 68.5 vs 65.4.
  • Perception Jumps: Models like Mipha-3B saw dramatic improvements in perception benchmarks like MMVP, jumping from 32.0 to 45.3 after incorporating ALLaVA data.

Conclusion and Future Directions

ALLaVA demonstrates that lite models can punch significantly above their weight when trained on high-quality, GPT-4V-distilled data. While the current dataset is primarily English-centric and synthetic data still requires refinement to eliminate all inaccuracies, the project provides a scalable path toward efficient, high-performance edge AI. The authors have open-sourced the dataset and models to encourage further development of resource-efficient multimodal systems.

Summary updated

Abstract

Recent advancements in Large Vision-Language Models (LVLMs) have enabled processing of multimodal inputs in language models but require significant computational resources for deployment, especially in edge devices. This study aims to bridge the performance gap between traditional-scale LVLMs and resource-friendly lite versions by adopting high-quality training data. To do this, a synthetic dataset is created by leveraging GPT-4V's ability to generate detailed captions, complex reasoning instructions and detailed answers from images. The resulted model trained with our data, ALLaVA, achieves competitive performance on 12 benchmarks up to 3B LVLMs. This work highlights the feasibility of adopting high-quality data in crafting more efficient LVLMs. Our online demo is available at \url{https://allava.freedomai.cn}.
BibTeX
@misc{chen2024allava,
  author = {Guiming Hardy Chen and Shunian Chen and Ruifei Zhang and Junying Chen and Xiangbo Wu and Zhiyi Zhang and Zhihong Chen and Jianquan Li and Xiang Wan and Benyou Wang},
  title = {{ALLaVA: Harnessing GPT4V-synthesized Data for A Lite Vision-Language Model}},
  year = {2024},
  eprint = {2402.11684},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2402.11684},
}
Figure