Introduction
Fine-tuning Multimodal Large Language Models (MLLMs) for specialized domains—such as dermatology, agriculture, or disaster response—significantly boosts their performance. However, the mechanism behind this improvement is counterintuitive. Conventional wisdom suggests that the "projection layer," which bridges vision and language, learns to extract better visual features during fine-tuning. This study reveals a surprising paradox: as the overall model improves, the projection layer actually becomes less informative on its own. This suggests that the LLM backbone, rather than the cross-modal bridge, is responsible for the heavy lifting of interpreting domain-specific visual attributes.
The Architecture and the Problem
Modern MLLMs like LLaVA typically consist of three components: a frozen vision encoder (e.g., CLIP), a large language model (LLM), and a projection layer (often a simple MLP) that aligns visual tokens with the LLM's textual embedding space.
When adapting these models to niche tasks, researchers typically use one of two fine-tuning strategies:
- FT-Proj: Updating only the projection layer while keeping the LLM frozen.
- FT-E2E: Updating both the projection layer and the LLM concurrently.
The core question is: which component learns the new visual concepts? Does the projection layer become a more sophisticated "translator" of visual features, or does the LLM simply become better at decoding the existing signals?
Experimental Methodology
To investigate this, the researchers introduced an independent MLP probe. This separate classifier was trained on the output of the projection layer to measure its "information richness." If the projection layer were learning richer domain-specific features, this probe's performance should improve post-fine-tuning. Conversely, if the probe's performance drops while the MLLM's performance rises, it indicates that the LLM is doing the cognitive work of "seeing" the new domain.
Datasets and Domains
The study evaluated performance across four distinct, specialized domains to test the limits of general-purpose vision encoders:
- Agriculture: Leaf disease classification using the PlantDoc dataset.
- Textures: Human-centric visual attributes from the Describable Textures Dataset (DTD).
- Dermatology: Identifying skin diseases via DermNet.
- Humanitarian: Classifying disaster-related imagery from CrisisMMD.
Key Insights: The Paradox of Information Richness
The Inverse Relationship
The most striking finding is the inverse relationship between MLLM performance and projection richness. In the Agriculture task, while the MLLM's F1 score jumped from 0.1064 to 0.5984 after end-to-end fine-tuning, the independent probe's performance on those same projections actually dropped by 6.22% to 27.49% depending on the setting. In Dermatology, the MLLM performance increased by a staggering 651%, yet the information richness in the projection layer decreased by nearly 19%.
The LLM as the Visual Processor
Even when the LLM is frozen (FT-Proj), it remains the primary component identifying domain-specific attributes. The projection layer does not become a better feature extractor; instead, it acts as a specialized "adapter" that rearranges data to better trigger the LLM's pre-existing internal knowledge. This suggests that the "bottleneck" in multimodal reasoning is not the vision encoder, but how the LLM chooses to attend to and interpret those features.
CLIP is Underutilized
Interestingly, raw CLIP features (pre-projection) often contain more useful information for specialized tasks than the MLLM actually utilizes. Simple image-only classifiers trained directly on CLIP embeddings outperformed fine-tuned MLLMs on three out of four tasks, highlighting a significant gap in how current multimodal architectures leverage available visual information.
Conclusion
These results challenge the prevailing view of the projection layer as a "feature aligner." Instead, the projection layer serves as a facilitator for the LLM's internal reasoning. Future MLLM research should shift focus away from simply improving the "bridge" between modalities and toward understanding how LLMs represent and reason over visual concepts internally. This shift is crucial for designing more efficient architectures and improving the interpretability of multimodal systems.
Summary updated