IAA: Inner-Adaptor Architecture Empowers Frozen Large Language Model with Multimodal Capabilities

Bin Wang, Chunyu Xie, Dawei Leng, Yuhui Yin

Abstract

In the field of multimodal large language models (MLLMs), common methods typically involve unfreezing the language model during training to foster profound visual understanding. However, the fine-tuning of such models with vision-language data often leads to a diminution of their natural language processing (NLP) capabilities. To avoid this performance degradation, a straightforward solution is to freeze the language model while developing multimodal competencies. Unfortunately, previous works have not attained satisfactory outcomes. Building on the strategy of freezing the language model, we conduct thorough structural exploration and introduce the Inner-Adaptor Architecture (IAA). Specifically, the architecture incorporates multiple multimodal adaptors at varying depths within the large language model to facilitate direct interaction with the inherently text-oriented transformer layers, thereby enabling the frozen language model to acquire multimodal capabilities. Unlike previous approaches of freezing language models that require large-scale aligned data, our proposed architecture is able to achieve superior performance on small-scale datasets. Our approach remarkably outperforms previous state-of-the-art methods across various vision-language benchmarks without sacrificing performance on NLP tasks.

Introduction

Most Multimodal Large Language Models (MLLMs), such as LLaVA, are constructed by unfreezing a Large Language Model (LLM) and fine-tuning it on image-text data. While this approach yields strong visual understanding, it frequently results in catastrophic forgetting—a significant degradation in the model's original natural language processing (NLP) capabilities. For instance, a model fine-tuned for vision might lose its proficiency in complex reasoning or coding tasks.

Performance degradation in standard MLLMs: LLaVA variants show significant drops in MMLU and C-eval scores compared to their base LLMs.

Existing solutions often involve freezing the LLM and training only a simple "connector" (projector). However, these methods typically fail to achieve high-level multimodal competence without massive, expensive datasets. The Inner-Adaptor Architecture (IAA) addresses this by providing a frozen LLM with deep visual capabilities without compromising its core reasoning "brain."

The Inner-Adaptor Architecture

IAA moves beyond simple input-level alignment by integrating visual information directly into the LLM's internal layers. The architecture is defined by several key innovations:

Dual-Workflow Design

To ensure zero performance loss on text-only tasks, IAA maintains two distinct paths. The Text-only Workflow utilizes the original frozen layers exclusively. The Multimodal Workflow routes data through both the frozen transformer layers and newly introduced trainable "insertion layers."

The IAA framework features a dual-workflow design, allowing the frozen LLM to handle text-only tasks while the multimodal workflow utilizes trainable insertion layers.

Insertion Layers (Inner-Adaptors)

Instead of merely appending image tokens to the input, IAA inserts trainable adaptors at various depths throughout the model (e.g., at layers 8 or 14). These layers are initialized using the weights of the frozen LLM layers they follow, ensuring the adaptors "speak the same language" as the base model from the start. The authors found that a sequential approach—where the output of a frozen layer serves as the input to an adaptor—outperformed additive, ControlNet-style architectures.

Specialized Components

To further prevent interference between modalities, the architecture introduces a dedicated multimodal embedding layer ($EL_{mm}$) and a language model head ($LH_{mm}$), keeping the original text-only components entirely untouched.

Key Results and Insights

IAA demonstrates that multimodal intelligence does not require sacrificing NLP prowess.

  • Preserving NLP Performance: While standard LLaVA-Llama3 scores on the MMLU benchmark dropped from 66.6 to 55.8, the IAA-8 configuration maintained a score of 68.4, effectively preserving (and slightly enhancing) the base model's reasoning capabilities.
  • Superior Multimodal Competence: Despite being trained on a relatively small dataset (1.2M–1.5M samples), IAA-14 outperformed LLaVA-Llama3 on benchmarks like MME (1591.5 vs 1506.0) and MMMU (39.9 vs 36.8).
  • Visual Grounding: IAA achieved state-of-the-art results on RefCOCO benchmarks, demonstrating superior object localization compared to models trained on significantly larger grounding datasets.

Conclusion and Future Work

The Inner-Adaptor Architecture provides a robust framework for empowering frozen LLMs with multimodal capabilities. While the additional layers introduce a minor increase in inference time (from 0.103s to 0.124s per token) and memory footprint, the benefits of preserving NLP performance are substantial. Future research may extend this "Inner-Adaptor" philosophy to other modalities, such as audio and video, to create a universal frozen multimodal assistant.

Summary updated

Abstract

In the field of multimodal large language models (MLLMs), common methods typically involve unfreezing the language model during training to foster profound visual understanding. However, the fine-tuning of such models with vision-language data often leads to a diminution of their natural language processing (NLP) capabilities. To avoid this performance degradation, a straightforward solution is to freeze the language model while developing multimodal competencies. Unfortunately, previous works have not attained satisfactory outcomes. Building on the strategy of freezing the language model, we conduct thorough structural exploration and introduce the Inner-Adaptor Architecture (IAA). Specifically, the architecture incorporates multiple multimodal adaptors at varying depths within the large language model to facilitate direct interaction with the inherently text-oriented transformer layers, thereby enabling the frozen language model to acquire multimodal capabilities. Unlike previous approaches of freezing language models that require large-scale aligned data, our proposed architecture is able to achieve superior performance on small-scale datasets. We conduct extensive experiments to improve the general multimodal capabilities and visual grounding abilities of the MLLM. Our approach remarkably outperforms previous state-of-the-art methods across various vision-language benchmarks without sacrificing performance on NLP tasks. Code and models are available at https://github.com/360CVGroup/Inner-Adaptor-Architecture.
BibTeX
@inproceedings{wang2025iaa,
  author = {Bin Wang and Chunyu Xie and Dawei Leng and Yuhui Yin},
  title = {{IAA: Inner-Adaptor Architecture Empowers Frozen Large Language Model with Multimodal Capabilities}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  url = {https://openreview.net/forum?id=V0KZh5wRG9},
}
Figure