Multi-modal Auto-regressive Modeling via Visual Tokens

Tianshuo Peng, Zuchao Li, Lefei Zhang, Hai Zhao, Ping Wang, Bo Du

TL;DR

This paper introduces "visual tokens" to enable true auto-regressive modeling for multi-modal Large Language Models (LLMs). By mapping visual features into the LLM's existing vocabulary space, they achieve a unified objective for both language and visual understanding, leading to improved performance and more efficient use of training data.

What Problem Does This Paper Address?

Large Language Models (LLMs) excel due to their auto-regressive training on vast text corpora, allowing them to learn general reasoning and perceptual abilities. Extending this to multi-modal models (LMMs) that incorporate images has been challenging. Current LMMs often treat visual information as continuous embeddings, which don't naturally fit into the discrete, token-based auto-regressive framework of LLMs. This leads to a disconnect: visual information is used as context but doesn't provide direct supervision for visual learning, resulting in under-utilized training data and limitations in true multi-modal comprehension. Existing methods either focus solely on language prediction or use separate visual regression/tokenization schemes that don't fully integrate with the LLM's core auto-regressive mechanism.

Methods

The core innovation is the concept of visual tokens. Instead of using raw visual embeddings, the authors propose a "VM head" (Visual Modeling head) that maps visual features into probability distributions over the LLM's vocabulary. These distributions are termed "visual tokens." This approach allows visual information to be represented as discrete tokens, similar to text tokens, enabling them to be seamlessly integrated into the LLM's auto-regressive generation process.

The proposed model, VT-LMM (Visual Token guided Large Multi-modal Model), has a framework consisting of:

  • Visual Encoder: Processes input images.
  • Adapter: Projects visual features into the LLM's semantic space.
  • Multi-modal Decoder & MM Head: Inherited from pre-trained LLMs, responsible for language generation and multi-modal reasoning.
  • VM Head: The novel component that generates visual tokens from visual features.

The training involves a multi-stage process:

  1. Stage I (Alignment): Adapter is trained to align visual features with the LLM's semantic space using image captioning.
  2. Stage II (Comprehension): The model is trained on diverse multi-modal instruction data for general comprehension.
  3. Stage III (Visual Tokenization): The VM head is trained to map visual features to visual tokens, using a classification-like loss.
  4. Stage IV (Unified Training): The full VT-LMM is trained with a unified objective, combining language and visual token prediction.

Additionally, the paper explores using fused text embeddings as pseudo image features. This involves using the generated visual tokens and the LLM's pre-trained text embeddings to construct alternative visual inputs, testing the hypothesis that the LLM's embedding space can inherently represent visual information.

Datasets

The paper utilizes several datasets for training and evaluation:

  • Training: LLaVA 1.5 (558k and mix-665k), LCS-558K (images).
  • Evaluation:
    • Visual Question Answering (VQA) Benchmarks: VQAv2, GQA, VisWiz, ScienceQA-IMG, TextVQA.
  • Benchmark Toolkits: POPE, MMBench, MMBench-Chinese, MM-Vet.

Key Insights and Ideas

  • Visual Tokens as a Bridge: The core idea that visual features can be mapped to the LLM's discrete vocabulary space is a significant conceptual leap. It allows for a unified auto-regressive objective, treating vision and language more symmetrically.
  • Unified Objective: Moving beyond separate language prediction losses or regression tasks for vision, the visual token approach enables a single, classification-based objective for both modalities. This is crucial for efficient and effective multi-modal learning.
  • Interpretable Visual Semantics: The visualization of visual tokens reveals that they can capture low-level features (color, contours) and higher-level semantics (object categories, actions), demonstrating that the LLM's semantic space can indeed represent visual information.
  • Pseudo Image Features: The success of using fused text embeddings as pseudo image features suggests that LLM's embedding layer is highly capable of representing visual information, further validating the semantic mapping achieved by visual tokens.

Key Results

  • Performance Gains: VT-LMM achieves state-of-the-art or competitive performance across multiple VQA benchmarks and toolkits, often outperforming models of similar scale (7B parameters) and even larger models (13B, 33B). For instance, VT-LMM-Mistral-7B shows strong results on object hallucination evaluation (Table 4) and general VQA tasks (Table 1, Table 5).
  • Efficiency: By leveraging the LLM's existing vocabulary and embedding space, VT-LMM avoids introducing massive external visual tokenizers or complex architectures, making it more parameter-efficient.
  • Improved Consistency: The unified auto-regressive modeling leads to better vision-language consistency, as evidenced by improved object hallucination evaluation.
  • Ablation Studies: Ablation studies confirm that the introduction of visual tokens as supervision significantly enhances performance across various LLM backbones (Vicuna-7B, Mistral-7B), validating their effectiveness.

Limitations and Future Work

  • Information Loss: Mapping continuous visual information to discrete tokens inherently involves some information loss, as suggested by the slight performance drop when using pseudo image features.
  • Diversity of Training Data: The impact of training image diversity on the VM head's learning effectiveness was not fully explored.
  • Future Directions: The authors suggest exploring more appropriate structures for constructing visual tokens and verifying the learning effect of the VM head with richer image datasets. Further investigation into how to better accomplish the projection of visual information into the language semantic space is also planned.

Summary updated

Abstract

Large Language Models (LLMs), benefiting from the auto-regressive modelling approach performed on massive unannotated texts corpora, demonstrates powerful perceptual and reasoning capabilities. However, as for extending auto-regressive modelling to multi-modal scenarios to build Large Multi-modal Models (LMMs), there lies a great difficulty that the image information is processed in the LMM as continuous visual embeddings, which cannot obtain discrete supervised labels for classification. In this paper, we successfully perform multi-modal auto-regressive modeling with a unified objective for the first time. Specifically, we propose the concept of visual tokens, which maps the visual features to probability distributions over LLM's vocabulary, providing supervision information for visual modelling. We further explore the distribution of visual features in the semantic space within LMM and the possibility of using text embeddings to represent visual information. Experimental results and ablation studies on 5 VQA tasks and 4 benchmark toolkits validate the powerful performance of our proposed approach.
BibTeX
@inproceedings{peng2024multi,
  author = {Tianshuo Peng and Zuchao Li and Lefei Zhang and Hai Zhao and Ping Wang and Bo Du},
  title = {{Multi-modal Auto-regressive Modeling via Visual Tokens}},
  booktitle = {ACM Multimedia},
  year = {2024},
  doi = {10.1145/3664647.3681685},
}

Loading related content...

Figure