Multi-modal Auto-regressive Modeling via Visual Words

Tianshuo Peng, Zuchao Li, Lefei Zhang, Hai Zhao, Ping Wang, Bo Du

Abstract

Large Language Models (LLMs), benefiting from the auto-regressive modelling approach performed on massive unannotated texts corpora, demonstrates powerful perceptual and reasoning capabilities. However, as for extending auto-regressive modelling to multi-modal scenarios to build Large Multi-modal Models (LMMs), there lies a great difficulty that the image information is processed in the LMM as continuous visual embeddings, which cannot obtain discrete supervised labels for classification. In this paper, we successfully perform multi-modal auto-regressive modeling with a unified objective for the first time. Specifically, we propose the concept of visual tokens, which maps the visual features to probability distributions over LLM's vocabulary, providing supervision information for visual modelling. We further explore the distribution of visual features in the semantic space within LMM and the possibility of using text embeddings to represent visual information. Experimental results and ablation studies on 5 VQA tasks and 4 benchmark toolkits validate the powerful performance of our proposed approach.

Introduction

While Large Language Models (LLMs) excel at processing discrete text tokens, extending them to multi-modal scenarios (LMMs) is challenging. Currently, most LMMs treat images as continuous visual embeddings, creating a fundamental "modality gap." This gap manifests in two primary ways:

  1. The Supervision Gap: Continuous vectors lack discrete labels, preventing models from being trained to "predict" images via classification in the same way they predict text.
  2. The Training Imbalance: Most models use images as contextual hints for text generation rather than learning the underlying distribution of visual data itself.

Existing solutions often rely on complex regression tasks or external image tokenizers (like VQ-VAE) that do not share the LLM's semantic space.

Methodology

The proposed framework, VT-LMM (Visual Token guided Large Multi-modal Model), introduces a Visual Modeling (VM) head. Instead of a new visual vocabulary, the VM head maps visual features directly onto the LLM’s existing text vocabulary. Each image patch is assigned a probability distribution over known words, effectively "labeling" image parts with text-based semantic tokens.

Comparison of visual modeling methods: (a) Visual Context, (b) Visual Regression, and (c) the proposed Multi-modal Auto-regressive Method.

The training process involves four distinct stages: alignment of visual features, instruction tuning on multi-modal tasks, VM head training to translate features into vocabulary distributions, and finally, unified multi-modal modeling using a classification objective.

Experimental Setup

VT-LMM was trained on high-quality open-source datasets including LLaVA 1.5-558k for alignment and LLaVA 1.5-mix-665k for instruction tuning. Evaluation spanned five VQA tasks (VQAv2, GQA, VisWiz, ScienceQA-IMG, and TextVQA) and four benchmark toolkits (POPE, MMBench, MMBench-Chinese, and MM-Vet).

Results and Performance

VT-LMM demonstrates that architectural efficiency can outperform raw scale. The 7B parameter version of VT-LMM (referred to as VW-LLM in performance charts) outperformed several 13B models and rivaled 33B models on benchmarks like GQA and VQAv2.

Performance comparison of VT-LMM (VW-LLM) against other state-of-the-art LMMs across nine benchmarks.

Key performance highlights include:

  • SOTA Results: Achieving a score of 44.0 on MM-Vet using the Mistral-7B backbone.
  • Hallucination Reduction: On the POPE benchmark, VT-LMM showed a significant reduction in object hallucination, as the model is forced to model the visual distribution explicitly rather than guessing based on text patterns.

Discussion: Visual Features in Semantic Space

A key insight of this work is that visual features in a trained LMM naturally reside in a semantic space close to text. For instance, image patches of a parrot's face align with tokens like "hair" or "feather." Furthermore, experiments with "pseudo-image features"—reconstructed entirely from text embeddings—showed that LLM text embeddings are sufficiently complete to represent complex visual information.

Conclusion

VT-LMM successfully bridges the modality gap by treating images as "visual words" within the LLM's existing vocabulary. While mapping continuous data to discrete tokens can cause some information loss, the unified auto-regressive objective significantly enhances performance and reasoning. Future work will explore more sophisticated VM head architectures and the impact of increased training data diversity.

Summary updated

Abstract

Large Language Models (LLMs), benefiting from the auto-regressive modelling approach performed on massive unannotated texts corpora, demonstrates powerful perceptual and reasoning capabilities. However, as for extending auto-regressive modelling to multi-modal scenarios to build Large Multi-modal Models (LMMs), there lies a great difficulty that the image information is processed in the LMM as continuous visual embeddings, which cannot obtain discrete supervised labels for classification.In this paper, we successfully perform multi-modal auto-regressive modeling with a unified objective for the first time.Specifically, we propose the concept of visual tokens, which maps the visual features to probability distributions over LLM's vocabulary, providing supervision information for visual modelling.We further explore the distribution of visual features in the semantic space within LMM and the possibility of using text embeddings to represent visual information.Experimental results and ablation studies on 5 VQA tasks and 4 benchmark toolkits validate the powerful performance of our proposed approach.
BibTeX
@inproceedings{peng2024multi,
  author = {Tianshuo Peng and Zuchao Li and Lefei Zhang and Hai Zhao and Ping Wang and Bo Du},
  title = {{Multi-modal Auto-regressive Modeling via Visual Words}},
  booktitle = {ACM Multimedia},
  year = {2024},
  doi = {10.1145/3664647.3681685},
}
Figure