Reformulating Vision-Language Foundation Models and Datasets Towards Universal Multimodal Assistants

Tianyu Yu, Jinyi Hu, Yuan Yao, Haoye Zhang, Yue Zhao, Chongyi Wang, Shan Wang, Yinxv Pan, Jiao Xue, Dahai Li, Zhiyuan Liu, Hai-Tao Zheng, Maosong SunarXiv 2023 · DOI · OpenReview

Introduction

The quest for a "Universal Multimodal Assistant"—an AI capable of seamless visual perception, reasoning, and dialogue—faces two primary hurdles: architectural alignment and data scarcity. Current models often rely on computationally expensive "bridge" modules to connect vision encoders with Large Language Models (LLMs), or they suffer from "knowledge scarcity" due to simplistic instruction-tuning datasets that lack deep context.

This paper introduces Muffin, a novel Multimodal Large Language Model (MLLM) framework that leverages pre-trained vision-language models (VLMs) as efficient connectors. Alongside Muffin, the authors present UniMM-Chat, a massive 1.1-million-instruction dataset designed to provide deep, knowledge-intensive context for multimodal training.

The Muffin Framework: VLMs as Efficient Bridges

Traditional MLLM architectures use linear layers or complex Q-Formers to align visual features with text embeddings. Muffin simplifies this by using a pre-trained VLM (specifically BEiT-3) as an "out-of-the-box" connector.

The Muffin architecture utilizes a pre-trained VLM to fuse visual and textual information before passing it to the LLM.

As shown in the architecture diagram, Muffin introduces trainable "query vectors" into the VLM’s text embedding space. These queries undergo deep cross-attention with image patches within the VLM's textual and visual channels. The resulting language-aligned visual features are then projected into the embedding space of a Vicuna-13B LLM, serving as a visual prefix for the conversation. This approach bypasses the need for resource-heavy bridge pre-training by capitalizing on the pre-existing alignment within foundation VLMs.

UniMM-Chat: Merging Complementary Knowledge

To address data scarcity, the authors developed the UniMM-Chat pipeline. Instead of treating different datasets as isolated sources, they merged annotations from five distinct sources—including COCO Captions, VQAv2, OKVQA, A-OKVQA, and Visual Dialog—for the same set of images.

By prompting ChatGPT to synthesize these "super-annotations" into multi-turn dialogues, the authors created a dataset that is significantly more knowledge-intensive. For instance, a simple image of a plane is enriched with its specific airline, the season (deduced from cherry blossoms), and the reasoning behind these identifications. This process yielded 1.1 million high-quality instructions across 117,000 dialogues.

Performance and Results

Muffin demonstrates state-of-the-art performance across a wide range of benchmarks. On the newly introduced UniMM-Bench, which focuses on reasoning and world knowledge, Muffin achieved an average score of 73.4, outperforming LLaVA (67.2) and InstructBLIP (68.8).

Muffin outperforms existing models across various visual reasoning and conversation benchmarks.

The radar chart highlights Muffin's superior capabilities in complex reasoning, conversation, and detailed description compared to other leading models like LLaVA and mPLUG-owl. Notably, Muffin scored 85.7 on the LLaVA test set, a significant improvement over the original LLaVA model. Human evaluations further confirm these findings, with Muffin being preferred for its helpfulness and correctness.

Key Insights and Future Directions

A critical finding of this work is that high image resolution (up to 672 pixels) is essential for tasks requiring fine-grained detail, such as reading text or identifying small objects. Furthermore, the success of UniMM-Chat proves that cross-dataset context is more effective than simply mixing datasets without integration.

While Muffin is highly effective, it remains tethered to the underlying LLM's capabilities and requires substantial resources for high-resolution tuning. Future research will explore scaling this "out-of-the-box" bridge concept with different combinations of VLMs and LLMs to further push the boundaries of universal multimodal assistance.

Summary updated

Abstract

Recent Multimodal Large Language Models (MLLMs) exhibit impressive abilities to perceive images and follow open-ended instructions. The capabilities of MLLMs depend on two crucial factors: the model architecture to facilitate the feature alignment of visual modules and large language models; the multimodal instruction tuning datasets for human instruction following. (i) For the model architecture, most existing models introduce an external bridge module to connect vision encoders with language models, which needs an additional feature-alignment pre-training. In this work, we discover that compact pre-trained vision language models can inherently serve as ``out-of-the-box'' bridges between vision and language. Based on this, we propose Muffin framework, which directly employs pre-trained vision-language models to act as providers of visual signals. (ii) For the multimodal instruction tuning datasets, existing methods omit the complementary relationship between different datasets and simply mix datasets from different tasks. Instead, we propose UniMM-Chat dataset which explores the complementarities of datasets to generate 1.1M high-quality and diverse multimodal instructions. We merge information describing the same image from diverse datasets and transforms it into more knowledge-intensive conversation data. Experimental results demonstrate the effectiveness of the Muffin framework and UniMM-Chat dataset. Muffin achieves state-of-the-art performance on a wide range of vision-language tasks, significantly surpassing state-of-the-art models like LLaVA and InstructBLIP. Our model and dataset are all accessible at https://github.com/thunlp/muffin.
BibTeX
@misc{yu2023reformulating,
  author = {Tianyu Yu and Jinyi Hu and Yuan Yao and Haoye Zhang and Yue Zhao and Chongyi Wang and Shan Wang and Yinxv Pan and Jiao Xue and Dahai Li and Zhiyuan Liu and Hai-Tao Zheng and Maosong Sun},
  title = {{Reformulating Vision-Language Foundation Models and Datasets Towards Universal Multimodal Assistants}},
  year = {2023},
  eprint = {2310.00653},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2310.00653},
}
Figure