Decoupled Two-Stage Multimodal Alignment

This direction is merged into another direction: two-stage multimodal training. Its last version stays readable and citable here.

Version 1 of 139 papers40 conceptsfrozen 2026-10-04

When building a Multimodal Large Language Model (MLLM), the most intuitive approach might be to connect a vision encoder to a language model and train the whole system end-to-end. However, researchers have found that this often leads to "gradient dominance," where the high-capacity language model overwhelms the visual branch, or "language prior conflict," where the model ignores the image and simply guesses based on text patterns.

To solve this, a consensus has emerged across the field: Decoupled Two-Stage Alignment. Instead of learning everything at once, models first learn to "see" by aligning visual features to a language space, and only then learn to "reason" or follow instructions.

Stage One: Building the Bridge

The first stage of this paradigm focuses exclusively on translation. The goal is not to solve a task, but to map features from a foreign modality (like images, 3D CT scans, or motion sequences) into the "latent language" of a pre-trained Large Language Model (LLM).

In this phase, researchers typically freeze both the vision encoder and the LLM. Only a small "bridge" or "projection layer" is trained. As Ruipu Luo and colleagues demonstrate in Valley: Video Assistant with Large Language model Enhanced abilitY, this ensures the projection layer learns a stable mapping without the LLM shifting its internal representations to compensate for noisy initial visual inputs.

This strategy is particularly vital in specialized domains. For instance, in BiMediX2: Bio-Medical EXpert LMM for Diverse Medical Modalities, Sahal Shaji Mullappilly uses this stage to align complex medical imaging (like X-rays and CT scans) to a text-based embedding space. By keeping the backbone frozen, the model establishes a foundational understanding of what a "nodule" or "fracture" looks like in vector space before it ever tries to write a medical report.

Stage Two: Instruction Tuning and Adaptation

Once the bridge is built, the second stage introduces task-specific logic. Here, the model is fine-tuned on "instruction-following" data—pairs of inputs and desired outputs that teach the model how to behave.

During this stage, the projection layer is usually updated alongside the language model, often using parameter-efficient methods like Low-Rank Adaptation (LoRA). Fuxiao Liu and the team behind MMC: Advancing Multimodal Chart Understanding with Large-scale Instruction Tuning use this second phase to refine the model’s ability to reason about abstract data charts. Because the visual-to-textual alignment was already established in stage one, the second stage can focus on the high-level logic of data extraction and reasoning.

Why Decoupling Works: Preventing Dominance

The primary reason this two-stage approach succeeds is that it manages the "capacity imbalance" between modalities. Zhigang Chen and the authors of Factorized Learning Assisted with Large Language Model for Gloss-free Sign Language Translation refer to this as "Factorized Learning." They argue that if you train a powerful LLM and a visual encoder simultaneously, the LLM’s gradients will dominate the update process, preventing the visual encoder from ever learning robust representations.

Furthermore, decoupling helps mitigate "hallucinations." In Decoupled Proxy Alignment: Mitigating Language Prior Conflict for Multimodal Alignment in MLLMs, Chenkun Tan explains that models often overfit to the linguistic style of the training captions. By separating the alignment of visual features from the learning of dataset-specific language styles, the model is forced to prioritize visually grounded information over mere textual probability.

Variations on the Theme

While the "Projector-then-LLM" sequence is common, researchers have adapted this decoupling for various goals:

  • Semantic Anchoring: In Preserving Cross-Modal Consistency for CLIP-based Class-Incremental Learning, Haoran Chen uses a two-stage process where one modality acts as a "stable anchor" for the other. This prevents the model from forgetting old classes when learning new ones.
  • Progressive Optimization: For complex tasks like e-commerce, Xiaohao Liu in Fine-tuning Multimodal Large Language Models for Product Bundling first fine-tunes the LLM on textual task patterns before freezing it to align multimodal embeddings. This ensures the model masters the "logic" of the task before trying to integrate visual or acoustic data.
  • Cultural Adaptation: Yue Gong and the CTA-Flux team use a two-stage strategy to bake Chinese cultural semantics into English-centric models, first aligning language embeddings and then fine-tuning on culturally specific visual distributions.

By treating multimodal learning as a sequence of alignment and then adaptation, these researchers have moved away from "black box" end-to-end training toward a more modular, controllable, and effective architecture for vision-language understanding.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_decoupled_two_stage_multimodal_alignment_dbfb2002,
  title = {Decoupled Two-Stage Multimodal Alignment},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/decoupled-two-stage-multimodal-alignment/rdr_dbfb20025cc448fca26e89333c55404a},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_decoupled_two_stage_multimodal_alignment_dbfb2002_v1,
  title = {Decoupled Two-Stage Multimodal Alignment},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/decoupled-two-stage-multimodal-alignment/rdr_dbfb20025cc448fca26e89333c55404a/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}