The Projection Bottleneck in Vision Language Models

This direction is merged into another direction: cross-modal information loss. Its last version stays readable and citable here.

Version 1 of 122 papers37 conceptsfrozen 2026-10-04

Modern Multimodal Large Language Models (MLLMs) are typically built as a "cascade" of three distinct components: a pre-trained vision encoder (like CLIP), a large language model (LLM) backbone, and a relatively small projection layer that bridges the two. While this modular design allows researchers to leverage the strengths of both vision and text models, recent evidence suggests that the projection layer acts as a critical information bottleneck, filtering out the very details required for complex reasoning.

The Reasoning Gap

A recurring observation across the literature is a "reasoning gap": models can often recognize individual objects but fail to use that information for higher-level tasks. In Can Vision-Language Models be a Good Guesser? Exploring VLMs for Times and Location Reasoning, S. Zenz and colleagues demonstrate this using the WikiTiLo benchmark. They found that while vision encoders successfully capture sociocultural cues—like specific architectural styles or traditional clothing—the generative MLLM fails to reason about the time or location of the image.

The intuition here is that the projection layer, tasked with aligning visual features to the LLM's textual embedding space, prioritizes general semantic features over the "fine-grained" or "localized" details necessary for contextual inference. As S. Zenz notes, this interface may strip away subtle but essential details, preventing the reasoning engine from ever "seeing" the evidence it needs.

The Alignment Paradox

One might assume that fine-tuning an MLLM on specialized data would "teach" the projection layer to extract better features. However, Gaurav Verma and Kartik Sharma discovered a surprising paradox in Mysterious Projections: Multimodal LLMs Gain Domain-Specific Visual Capabilities Without Richer Cross-Modal Projections.

When they fine-tuned models for domains like agriculture or medicine, the models' performance improved, but the projection layers actually became less representative of domain-specific attributes. This suggests that the LLM backbone is doing the heavy lifting of interpreting the visual signals, while the bridge itself remains a lossy, low-fidelity translator. The raw features in the vision encoder often contain more comprehensive information than the "aligned" features that eventually reach the LLM.

Spatial Blindness and Perception Bottlenecks

The bottleneck effect is particularly damaging for spatial and fine-grained tasks. Zhiyuan Fan and the authors of V2R-Bench show that even top-tier models like GPT-4o are fragile to basic visual changes in position or scale. They argue that these vulnerabilities are rooted in the multimodal projector, which fails to transmit precise spatial data even when the underlying vision encoder captures it perfectly.

This "perception bottleneck" is further evidenced by the work of Ondrej Bohdal in VL-ICL Bench. Bohdal found that MLLMs perform significantly better on complex reasoning tasks when the visual input is replaced with a detailed text description of the scene. This implies that the model's reasoning engine is capable, but the visual interface is failing to provide the necessary granularity. Similarly, Gregor Geigle found that MLLMs often perform worse at fine-grained classification (e.g., identifying specific bird species) than the standalone CLIP encoders they are built upon, a phenomenon he calls the "alignment gap."

Why the Bottleneck Persists

Several factors contribute to this loss of information:

  1. Token Compression: To maintain efficiency, models often aggressively reduce the number of visual tokens. Song Han and the NVILA team point out that this compression makes alignment harder to train and often leads to a loss in reasoning capabilities.
  2. Training Data Mismatch: The projection layer is typically trained on much smaller datasets than the massive vision and language models it connects. Raehyuk Jung suggests this limits the bridge's ability to handle concepts not explicitly present in the alignment data.
  3. Linguistic Bias: As visual information moves through the model, intermediate layers often retain more precise details than the final output. Yaoyuan Liang argues that the final layers of MLLMs prioritize linguistic coherence and general patterns over granular visual truth, leading to hallucinations or omissions.

Measuring the Loss

To diagnose these issues, researchers have moved toward "parameter-free" metrics. Yueqian Wang introduced Pfram, a metric that measures how well internal representations align with ground-truth labels at the exact point where visual tokens are projected into the LLM. By isolating this interface, researchers can confirm that hallucinations often stem not from the LLM "imagining" things, but from the projection layer failing to maintain visual fidelity during the modality hand-off.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_the_projection_bottleneck_in_vision_lang_0323fd5a,
  title = {The Projection Bottleneck in Vision Language Models},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/the-projection-bottleneck-in-vision-language-models/rdr_0323fd5a4894468d824de401f4705bb0},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_the_projection_bottleneck_in_vision_lang_0323fd5a_v1,
  title = {The Projection Bottleneck in Vision Language Models},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/the-projection-bottleneck-in-vision-language-models/rdr_0323fd5a4894468d824de401f4705bb0/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}