Efficiently Integrate Large Language Models with Visual Perception: A Survey from the Training Paradigm Perspective

Xiaorui Ma, Haoran Xie, S. Joe Qin

TL;DR

This paper provides a comprehensive roadmap for transforming "blind" Large Language Models (LLMs) into Vision Large Language Models (VLLMs). By categorizing 34 major models into three distinct training paradigms—Single-stage, Two-stage, and Direct Adaptation—the authors highlight how researchers are balancing the hunger for multimodal reasoning with the need for parameter efficiency.

What Problem Does This Paper Address?

While LLMs like LLaMA and GPT-4 excel at text-based reasoning, they are natively "blind." Early attempts to bridge vision and language relied on Vision-Language Pretrained Models (VLPMs), which were good at captioning but lacked deep reasoning and required expensive fine-tuning for every new task.

The challenge today is integration efficiency. As LLMs scale to tens or hundreds of billions of parameters, retraining them to "see" is computationally prohibitive. This paper addresses the gap in our understanding of the different training "recipes" (paradigms) used to connect vision encoders to LLMs without breaking the bank or losing the LLM's hard-won reasoning capabilities.

Methods

The authors break down the VLLM architecture into three components: a Vision Encoder (VE), a Modality Integrator (MI), and the LLM itself. The core of the survey is the taxonomy of three training paradigms:

  1. Single-stage Tuning: Focuses on training only the Modality Integrator (the bridge) to align visual features with the LLM's text space. It’s efficient but often struggles with complex instructions.
  2. Two-stage Tuning: The current "gold standard" (e.g., LLaVA). Stage 1 aligns features; Stage 2 performs Instruction Tuning, often using Parameter-Efficient Adaptation (PEA) like LoRA to update the LLM's behavior without retraining all its weights.
  3. Direct Adaptation: The "budget-friendly" newcomer. It skips the expensive pretraining stage entirely, directly fine-tuning the Modality Integrator on downstream tasks using multi-task learning.

The paper also categorizes Modality Integrators into "Out-of-block" (external connectors like MLPs or Q-Formers) and "In-block" (modules inserted directly into the LLM's transformer layers, like bottleneck adapters).

Datasets

The survey reviews a vast array of datasets used across the field:

  • Traditional Benchmarks: VQAv2 (Visual Question Answering), GQA, and COCO Captions for basic perception.
  • Advanced/Instructional: MMBench, SEED-I, MM-Vet, and LLaVA-Bench, which test "hallucination" levels and complex reasoning.
  • Pretraining Data: Massive image-text pair collections like LAION-5B, CC3M, and SBU.

Key Insights and Ideas

  • The "Aha" of Direct Adaptation: Perhaps the most surprising insight is that the pretraining stage—long thought essential for alignment—can sometimes be skipped. By using multi-task learning, models can achieve competitive results with significantly less compute.
  • Information Bottlenecks: The authors note a trade-off in Modality Integrator design. Attention-based abstractors (like the Resampler) are great at shortening visual sequences for efficiency but can "lose" fine-grained details by focusing only on major objects.
  • The Rise of Small LLMs: There is a notable trend toward using "manageable" LLMs (1.4B to 7B parameters) like MobileVLM or Mamba-based architectures to bring VLLM capabilities to edge devices.
  • PEA is Mandatory: Parameter-Efficient Adaptation (LoRA, QLoRA, Adapters) is no longer an optional optimization; it is the primary way the community handles the "memory wall" of modern LLMs.

Key Results

  • Two-stage Dominance: Models like CogVLM and LLaVA 1.5 currently lead the pack in advanced reasoning benchmarks (MM-Vet and MME), proving that instruction tuning is vital for "human-like" interaction.
  • Efficiency Winners: The model MAPL (Single-stage) demonstrated extreme efficiency, training only 0.05% of total parameters while remaining competitive in captioning tasks.
  • Replicated Findings: In their own replication of the Direct Adaptation paradigm, the authors found that Bottleneck Adapters (like those in VL-PET) often provide a better balance of performance vs. parameter count than the more popular LoRA method.

Limitations and Future Work

  • Fine-grained Perception: Most current VLLMs still struggle with very small details or dense text within images (OCR), as the Modality Integrators often compress visual data too aggressively.
  • Computational Costs: Despite efficiency gains, training even a "small" 7B VLLM remains out of reach for many academic labs without significant GPU clusters.
  • Future Direction: The authors suggest that the next frontier is applying these efficient MI designs to even larger "frontier" models and improving the "Direct Adaptation" paradigm to match the instruction-following prowess of two-stage models.

Summary updated

BibTeX
@article{ma2026efficiently,
  author = {Xiaorui Ma and Haoran Xie and S. Joe Qin},
  title = {{Efficiently Integrate Large Language Models with Visual Perception: A Survey from the Training Paradigm Perspective}},
  journal = {Information Fusion},
  year = {2026},
  doi = {10.1016/j.inffus.2025.103419},
}
Figure