Efficiently Integrate Large Language Models with Visual Perception: A Survey from the Training Paradigm Perspective

Xiaorui Ma, Haoran Xie, S. Joe Qin

Abstract

The integration of visual perception into Large Language Models (LLMs) has evolved rapidly, shifting from end-to-end training to the efficient adaptation of off-the-shelf models. This survey categorizes the evolution of Vision Large Language Models (VLLMs) into three distinct training paradigms: Single-stage Tuning, Two-stage Tuning, and Direct Adaptation. Through an analysis of 34 models, we explore the balance between computational efficiency and multimodal performance. While Two-stage Tuning currently achieves state-of-the-art results, Direct Adaptation emerges as a highly efficient and increasingly competitive alternative for resource-constrained environments.

Introduction

Historically, multimodal tasks were addressed by Vision-Language Pretrained Models (VLPMs) trained from scratch. However, the emergence of massive, high-performing text-only LLMs has shifted the paradigm toward "bolting on" vision encoders to existing models. Given the scale of modern LLMs, full-parameter fine-tuning is often computationally prohibitive. Researchers have turned to Parameter-Efficient Adaptation (PEA), which involves training only a small fraction of the model to bridge the gap between visual and textual modalities. This survey fills a critical gap by analyzing the evolution of these training strategies and their impact on reasoning and perception performance.

Taxonomy of Training Paradigms

The core of a VLLM architecture typically includes a frozen LLM, a frozen Vision Encoder (VE), and a Modality Integrator (MI). The MI serves as the bridge, translating visual features into the LLM's embedding space. These integrators are classified into Out-of-block Integrators (e.g., linear projectors, Q-Formers) and In-block Integrators (e.g., bottleneck adapters within transformer layers).

The training process is categorized into three main paradigms:

  1. Single-stage Tuning: The MI is pretrained on large-scale image-text pairs to align modalities. Downstream performance relies on the LLM's inherent zero-shot capabilities.
  2. Two-stage Tuning: Following initial MI pretraining (Stage 1), the model undergoes Instruction Tuning (Stage 2). This stage uses formatted datasets to teach the model to follow human instructions and perform complex reasoning.
  3. Direct Adaptation: This paradigm skips the initial pretraining phase entirely. The MI is directly fine-tuned on specific downstream multi-task datasets, offering a highly resource-conscious approach.
The three primary training paradigms for VLLMs: Single-stage Tuning, Two-stage Tuning, and Direct Adaptation.

Datasets and Evaluation

The VLLM lifecycle utilizes diverse datasets across different stages:

  • Pretraining: Large-scale image-text pairs such as LAION-5B and Conceptual Captions (CC3M/12M) are used for initial alignment.
  • Instruction Tuning: Combines text-only conversational data (e.g., ShareGPT) with multimodal instruction data (e.g., LLaVA, MiniGPT4).
  • Evaluation: Benchmarks are split into Traditional (VQAv2, COCO) for basic perception and Advanced (MMBench, MME, MM-Vet) for complex reasoning and instruction-following, often employing GPT-4 as an automated evaluator.

Key Insights and Results

Analysis of the surveyed models reveals several critical trends:

  • Performance Leadership: Two-stage Tuning remains the dominant paradigm for high-end performance. Models like CogVLM-Chat and LLaVA 1.5 achieve top scores on advanced benchmarks (e.g., CogVLM-Chat scoring 77.6 on MMBench).
  • Efficiency of Direct Adaptation: Direct Adaptation is remarkably efficient, often achieving competitive scores on traditional benchmarks while training less than 8% of total parameters. Techniques like Ladder Side-Tuning (LST) can reduce training memory usage by over 50%.
  • The Power of Frozen LLMs: Updating the LLM is not always necessary. Models that train only the MI can achieve state-of-the-art results, preserving the LLM's original linguistic capabilities.
  • Architectural Opportunities: While attention-based integrators are common, convolution-based abstractors (e.g., MobileVLM) offer significant parameter efficiency and remain an underexplored area for lightweight design.

Conclusion and Future Directions

While Single-stage Tuning often lags due to limited task adaptation, Direct Adaptation shows that massive pretraining is not always a prerequisite for strong downstream performance. A promising future direction involves applying Direct Adaptation techniques to larger LLMs (e.g., LLaMA-3) and merging Instruction Tuning into the Direct Adaptation workflow. This could potentially combine the deep alignment of Two-stage models with the computational efficiency of Direct Adaptation, paving the way for more accessible and capable multimodal AI.

Summary updated

BibTeX
@article{ma2026efficiently,
  author = {Xiaorui Ma and Haoran Xie and S. Joe Qin},
  title = {{Efficiently Integrate Large Language Models with Visual Perception: A Survey from the Training Paradigm Perspective}},
  journal = {Information Fusion},
  year = {2026},
  doi = {10.1016/j.inffus.2025.103419},
}
Figure