When building a multimodal model—whether it connects text to images, audio, molecules, or graphs—a naive approach would be to fine-tune the entire network end-to-end on target tasks from day one. However, this creates a fundamental optimization conflict. Large Language Models (LLMs) possess massive pre-trained reasoning capabilities, while raw encoders extract features that speak an entirely different mathematical language. Forcing a network to learn cross-modal communication while simultaneously learning complex task logic causes the language model's gradients to dominate, crushing the encoder's representations or leading to catastrophic forgetting.
To resolve this, researchers have converged on a decoupled, two-stage training curriculum. Stage 1 focuses on representation alignment (teaching the model to perceive), while Stage 2 focuses on instruction tuning (teaching the model to reason and converse).
Stage 1: Bridging the Modality Gap
In the first stage, the goal is to map foreign features (pixels, audio waveforms, molecular graphs) into the language model's native embedding space. Crucially, the language model is typically kept frozen during this phase.
As shown by Lu et al. in An Empirical Study of Scaling Instruct-Tuned Large Multimodal Models, this stage uses simple, high-volume data like image-caption pairs to optimize a lightweight projection layer or adapter. For instance, in XrayGPT, a projection layer is pre-trained to map chest radiographs to medical text before any clinical dialogue is attempted.
By freezing the LLM, the network forces the adapter to do the heavy lifting of translation. This ensures that the new modality behaves like "soft text tokens" that the LLM can naturally ingest. This "factorized learning" approach, as articulated by Chen et al. in Factorized Learning Assisted with Large Language Model for Gloss-free Sign Language Translation, prevents the high-capacity language model from overwhelming the visual updates, establishing a stable visual-semantic mapping before introducing complex task logic.
Stage 1: Alignment (Perception)
[Raw Input] -> [Encoder] -> [Trainable Adapter] -> [Frozen LLM] -> [Simple Caption/Label]
Stage 2: Instruction Tuning (Reasoning)
[Raw Input] -> [Encoder] -> [Aligned Adapter] -> [Trainable LLM/LoRA] -> [Complex Response]
Stage 2: Task-Specific Instruction Tuning
Once the latent spaces are aligned, the model enters the second stage. Now that the LLM can "perceive" the incoming tokens, the focus shifts to complex reasoning, multi-turn dialogue, and task-specific execution. In this stage, the language model's weights are unfrozen or adapted using parameter-efficient methods like Low-Rank Adaptation (LoRA).
Sun et al. demonstrate this transition in GPT4RoI. Their model first learns fundamental object-attribute associations (Stage 1) before attempting multi-round conversational reasoning on specific regions of interest (Stage 2). Similarly, Kong et al. show in Audio Flamingo that unfreezing the language model backbone during the second stage of supervised fine-tuning is vital for adapting its internal logic to handle multi-turn dialogue and non-speech audio cues.
Why This Separation Works
Decoupling perception from reasoning works because it respects the optimization dynamics of neural networks:
- Gradient Stabilization: It isolates the optimization of the cross-modal bridge. If both the encoder and the LLM are updated simultaneously on complex tasks from the start, the LLM often finds text-only shortcuts to minimize loss, ignoring the multimodal inputs entirely.
- Preserving Foundational Knowledge: Forcing an encoder to adapt too quickly can destroy its pre-trained visual common sense. Alain Pagani notes in 3D Spatial Understanding in MLLMs: Disambiguation and Evaluation that an initial alignment stage followed by parameter-efficient fine-tuning allows the model to adapt to spatial reasoning without retraining the entire architecture, preventing catastrophic forgetting.
- Structural Efficiency: It allows models to learn layout and structure before content. For example, Roy Ganz demonstrates in DocVLM: Make Your VLM an Efficient Reader that aligning a language model with compressed textual and layout features first, followed by a second phase that reintroduces visual signals, allows the model to master document structure efficiently.
This two-stage paradigm is highly generalizable. The same core mechanism of separating representation alignment from instruction following is successfully utilized across diverse domains, including molecular graphs in LLaMo, user interaction histories in User-LLM, and vector graphics generation by Itsumi Saito in Sketch2Diagram.