Large Language Models (LLMs) are exceptional reasoning engines, but they are fundamentally blind. They operate entirely on discrete tokens—vectors residing in a highly specific, learned linguistic embedding space. Conversely, vision encoders (such as Vision Transformers) process pixels into continuous spatial feature maps.
Feeding raw visual features directly into an LLM fails because of an impedance mismatch: the language model has no frame of reference for the coordinate system, scale, or semantic distribution of the vision encoder's latent space. To make an LLM multimodal without retraining its billions of parameters from scratch, we must build a bridge that translates visual features into the exact vector space the LLM expects.
The Projector as a Geometric Translator
To connect a frozen vision encoder to a frozen language model, researchers introduce a learnable cross-modal feature projector. In its simplest form, this connector is a single linear layer or a Multi-Layer Perceptron (MLP).
Why can a simple geometric transformation map something as complex as an image into a text space? The intuition lies in the structure of contrastive pre-trained models like CLIP. These models already align images and text into a shared semantic neighborhood. However, a persistent "modality gap" remains—visual vectors and textual vectors occupy distinct clusters within that space.
As explored by Gandelsman in Interpreting CLIP's Image Representation via Text-Based Decomposition, high-dimensional visual representations can be decomposed into interpretable text-based directions. The projection layer acts as a geometric translator, performing the necessary rotation, scaling, and translation to shift these visual features directly into the LLM's native token embedding space. Once projected, the visual features are structurally indistinguishable from word embeddings to the language model.
[Raw Image] -> [Vision Encoder] -> [Visual Features]
│
(Projector Layer)
▼
[Text Prompt] -> [Token Embeddings] -> [Unified Token Space] -> [Frozen LLM]
From Visual Prefixes to Unified Interfaces
The paradigm of cross-modal projection has evolved from simple sequence appending to sophisticated multi-modal pipelines. In a foundational approach, Tsimpoukelli et al. in Multimodal Few-Shot Learning with Frozen Language Models demonstrated that a frozen language model could act as a multimodal learner simply by treating projected visual features as a "visual prefix." By prepending these continuous embeddings to standard text tokens, the LLM processes the image as if it were the opening words of a sentence.
Building on this prefix concept, modern architectures use projection to handle a wider array of tasks. Chen et al. in MiniGPT-v2: large language model as a unified interface for vision-language multi-task learning formulated diverse vision-language challenges—such as image captioning and visual grounding—as unified text-generation sequences. By mapping visual tokens directly into the language model's embedding space, the system treats pixel data as a specialized dialect that fits naturally into the model's text-generation pipeline.
To scale this training efficiently, Dong et al. in Scalable Vision Language Model Training via High Quality Data Curation highlight the importance of isolating the training of this neural mapping layer during early alignment stages. By keeping the vision encoder and language backbone frozen initially, the projector learns to stabilize the cross-modal bridge without disrupting or distorting the pre-learned representations of the larger models.
Unifying Modalities Before Projection
When expanding beyond static images to temporal data like video, a naive approach would use separate projection layers for each input type. However, managing multiple independent projection paths increases architectural complexity and forces the language model to reconcile disparate structural formats.
To solve this, Lin et al. in Video-LLaVA: Learning United Visual Representation by Alignment Before Projection demonstrate that aligning different visual modalities (such as images and video frames) into a shared representation space before passing them through a single, unified MLP projector is significantly more effective.
This early unification ensures that the language model receives a cohesive visual vocabulary. Whether an input originates from a static JPEG or a sequence of video frames, the projection layer maps it into the LLM token space using a consistent geometric transformation. This allows the language model's attention mechanisms to seamlessly weigh visual and textual tokens, unlocking deep multimodal reasoning with minimal computational overhead.