TL;DR
This paper introduces "visual tokens" to enable true auto-regressive modeling for multi-modal Large Language Models (LLMs). By mapping visual features into the LLM's existing vocabulary space, they achieve a unified objective for both language and visual understanding, leading to improved performance and more efficient use of training data.
What Problem Does This Paper Address?
Large Language Models (LLMs) excel due to their auto-regressive training on vast text corpora, allowing them to learn general reasoning and perceptual abilities. Extending this to multi-modal models (LMMs) that incorporate images has been challenging. Current LMMs often treat visual information as continuous embeddings, which don't naturally fit into the discrete, token-based auto-regressive framework of LLMs. This leads to a disconnect: visual information is used as context but doesn't provide direct supervision for visual learning, resulting in under-utilized training data and limitations in true multi-modal comprehension. Existing methods either focus solely on language prediction or use separate visual regression/tokenization schemes that don't fully integrate with the LLM's core auto-regressive mechanism.
Methods
The core innovation is the concept of visual tokens. Instead of using raw visual embeddings, the authors propose a "VM head" (Visual Modeling head) that maps visual features into probability distributions over the LLM's vocabulary. These distributions are termed "visual tokens." This approach allows visual information to be represented as discrete tokens, similar to text tokens, enabling them to be seamlessly integrated into the LLM's auto-regressive generation process.
The proposed model, VT-LMM (Visual Token guided Large Multi-modal Model), has a framework consisting of:
- Visual Encoder: Processes input images.
- Adapter: Projects visual features into the LLM's semantic space.
- Multi-modal Decoder & MM Head: Inherited from pre-trained LLMs, responsible for language generation and multi-modal reasoning.
- VM Head: The novel component that generates visual tokens from visual features.
The training involves a multi-stage process:
- Stage I (Alignment): Adapter is trained to align visual features with the LLM's semantic space using image captioning.
- Stage II (Comprehension): The model is trained on diverse multi-modal instruction data for general comprehension.
- Stage III (Visual Tokenization): The VM head is trained to map visual features to visual tokens, using a classification-like loss.
- Stage IV (Unified Training): The full VT-LMM is trained with a unified objective, combining language and visual token prediction.
Additionally, the paper explores using fused text embeddings as pseudo image features. This involves using the generated visual tokens and the LLM's pre-trained text embeddings to construct alternative visual inputs, testing the hypothesis that the LLM's embedding space can inherently represent visual information.
Datasets
The paper utilizes several datasets for training and evaluation:
- Training: LLaVA 1.5 (558k and mix-665k), LCS-558K (images).
- Evaluation:
- Visual Question Answering (VQA) Benchmarks: VQAv2, GQA, VisWiz, ScienceQA-IMG, TextVQA.
- Benchmark Toolkits: POPE, MMBench, MMBench-Chinese, MM-Vet.
Key Insights and Ideas
- Visual Tokens as a Bridge: The core idea that visual features can be mapped to the LLM's discrete vocabulary space is a significant conceptual leap. It allows for a unified auto-regressive objective, treating vision and language more symmetrically.
- Unified Objective: Moving beyond separate language prediction losses or regression tasks for vision, the visual token approach enables a single, classification-based objective for both modalities. This is crucial for efficient and effective multi-modal learning.
- Interpretable Visual Semantics: The visualization of visual tokens reveals that they can capture low-level features (color, contours) and higher-level semantics (object categories, actions), demonstrating that the LLM's semantic space can indeed represent visual information.
- Pseudo Image Features: The success of using fused text embeddings as pseudo image features suggests that LLM's embedding layer is highly capable of representing visual information, further validating the semantic mapping achieved by visual tokens.
Key Results
- Performance Gains: VT-LMM achieves state-of-the-art or competitive performance across multiple VQA benchmarks and toolkits, often outperforming models of similar scale (7B parameters) and even larger models (13B, 33B). For instance, VT-LMM-Mistral-7B shows strong results on object hallucination evaluation (Table 4) and general VQA tasks (Table 1, Table 5).
- Efficiency: By leveraging the LLM's existing vocabulary and embedding space, VT-LMM avoids introducing massive external visual tokenizers or complex architectures, making it more parameter-efficient.
- Improved Consistency: The unified auto-regressive modeling leads to better vision-language consistency, as evidenced by improved object hallucination evaluation.
- Ablation Studies: Ablation studies confirm that the introduction of visual tokens as supervision significantly enhances performance across various LLM backbones (Vicuna-7B, Mistral-7B), validating their effectiveness.
Limitations and Future Work
- Information Loss: Mapping continuous visual information to discrete tokens inherently involves some information loss, as suggested by the slight performance drop when using pseudo image features.
- Diversity of Training Data: The impact of training image diversity on the VM head's learning effectiveness was not fully explored.
- Future Directions: The authors suggest exploring more appropriate structures for constructing visual tokens and verifying the learning effect of the VM head with richer image datasets. Further investigation into how to better accomplish the projection of visual information into the language semantic space is also planned.
Summary updated