Abstract
The integration of visual perception into Large Language Models (LLMs) has evolved rapidly, shifting from end-to-end training to the efficient adaptation of off-the-shelf models. This survey categorizes the evolution of Vision Large Language Models (VLLMs) into three distinct training paradigms: Single-stage Tuning, Two-stage Tuning, and Direct Adaptation. Through an analysis of 34 models, we explore the balance between computational efficiency and multimodal performance. While Two-stage Tuning currently achieves state-of-the-art results, Direct Adaptation emerges as a highly efficient and increasingly competitive alternative for resource-constrained environments.
Introduction
Historically, multimodal tasks were addressed by Vision-Language Pretrained Models (VLPMs) trained from scratch. However, the emergence of massive, high-performing text-only LLMs has shifted the paradigm toward "bolting on" vision encoders to existing models. Given the scale of modern LLMs, full-parameter fine-tuning is often computationally prohibitive. Researchers have turned to Parameter-Efficient Adaptation (PEA), which involves training only a small fraction of the model to bridge the gap between visual and textual modalities. This survey fills a critical gap by analyzing the evolution of these training strategies and their impact on reasoning and perception performance.
Taxonomy of Training Paradigms
The core of a VLLM architecture typically includes a frozen LLM, a frozen Vision Encoder (VE), and a Modality Integrator (MI). The MI serves as the bridge, translating visual features into the LLM's embedding space. These integrators are classified into Out-of-block Integrators (e.g., linear projectors, Q-Formers) and In-block Integrators (e.g., bottleneck adapters within transformer layers).
The training process is categorized into three main paradigms:
- Single-stage Tuning: The MI is pretrained on large-scale image-text pairs to align modalities. Downstream performance relies on the LLM's inherent zero-shot capabilities.
- Two-stage Tuning: Following initial MI pretraining (Stage 1), the model undergoes Instruction Tuning (Stage 2). This stage uses formatted datasets to teach the model to follow human instructions and perform complex reasoning.
- Direct Adaptation: This paradigm skips the initial pretraining phase entirely. The MI is directly fine-tuned on specific downstream multi-task datasets, offering a highly resource-conscious approach.
Datasets and Evaluation
The VLLM lifecycle utilizes diverse datasets across different stages:
- Pretraining: Large-scale image-text pairs such as LAION-5B and Conceptual Captions (CC3M/12M) are used for initial alignment.
- Instruction Tuning: Combines text-only conversational data (e.g., ShareGPT) with multimodal instruction data (e.g., LLaVA, MiniGPT4).
- Evaluation: Benchmarks are split into Traditional (VQAv2, COCO) for basic perception and Advanced (MMBench, MME, MM-Vet) for complex reasoning and instruction-following, often employing GPT-4 as an automated evaluator.
Key Insights and Results
Analysis of the surveyed models reveals several critical trends:
- Performance Leadership: Two-stage Tuning remains the dominant paradigm for high-end performance. Models like CogVLM-Chat and LLaVA 1.5 achieve top scores on advanced benchmarks (e.g., CogVLM-Chat scoring 77.6 on MMBench).
- Efficiency of Direct Adaptation: Direct Adaptation is remarkably efficient, often achieving competitive scores on traditional benchmarks while training less than 8% of total parameters. Techniques like Ladder Side-Tuning (LST) can reduce training memory usage by over 50%.
- The Power of Frozen LLMs: Updating the LLM is not always necessary. Models that train only the MI can achieve state-of-the-art results, preserving the LLM's original linguistic capabilities.
- Architectural Opportunities: While attention-based integrators are common, convolution-based abstractors (e.g., MobileVLM) offer significant parameter efficiency and remain an underexplored area for lightweight design.
Conclusion and Future Directions
While Single-stage Tuning often lags due to limited task adaptation, Direct Adaptation shows that massive pretraining is not always a prerequisite for strong downstream performance. A promising future direction involves applying Direct Adaptation techniques to larger LLMs (e.g., LLaMA-3) and merging Instruction Tuning into the Direct Adaptation workflow. This could potentially combine the deep alignment of Two-stage models with the computational efficiency of Direct Adaptation, paving the way for more accessible and capable multimodal AI.
Summary updated