Imagine asking a large language model (LLM) a text-based question: "In which city is the Eiffel Tower located?" The model will instantly and accurately answer "Paris". Now, take the exact same underlying model, equip it with a vision encoder to create a Vision-Language Model (VLM), and show it a clear photograph of the Eiffel Tower with the question: "In which city is this object located?"
Paradoxically, the VLM is highly likely to fail. Work by Ashok et al. demonstrates that VLMs suffer a performance drop of over 50% when retrieving factual knowledge via visual cues compared to text-based identifiers. This occurs even when the model successfully identifies the object in the image and undeniably possesses the relevant factual knowledge in its text-only memory.
This phenomenon reveals a fundamental cross-modal factual retrieval bottleneck. It is not a failure of sight (perception) or a failure of memory (knowledge), but rather a failure of connection.
The Seeing-Knowing Paradox
To understand why this happens, we must first isolate where the failure occurs. One might assume that the model simply fails to recognize the image. However, recent evaluation benchmarks show a distinct decoupling between visual recognition and factual retrieval.
As Gu et al. observe, a VLM can frequently output the correct name of an entity when asked "What is this?", but fail entirely when asked to provide encyclopedic or factual information about it. This disconnect is particularly stark in culturally specific domains. For instance, Li et al. introduced a benchmark for fine-grained culinary culture, revealing that while text-based models excel at describing regional dishes, VLMs trail human performance by up to 41% when trying to link the visual representation of those same dishes to their cultural contexts. Similarly, Park et al. document a persistent gap between a model's abstract textual grasp of cultural concepts and its ability to ground that knowledge in specific visual artifacts.
The problem is not that the model lacks the knowledge base, as emphasized by Wong et al.; expanding the number of parameters or adding more text data does not inherently fix the model's ability to apply that knowledge to visual inputs.
The Two-Hop Problem and Computational Depth
Why does a visual token fail to trigger the same memory circuits as a text token? The answer lies in the internal mechanics of how transformers process information across their layers.
When an LLM receives the text token "Eiffel Tower", the symbolic identity of the object is available immediately in the very first layer. The model's internal factual memory circuits—which are primarily active in the early-to-middle layers of the network—are triggered instantly.
However, when a VLM receives raw visual tokens from an image encoder, it must perform a "two-hop" process:
- Hop 1 (Perception): Process the visual features to resolve what the object is.
- Hop 2 (Retrieval): Use that identity to trigger associated factual knowledge.
Text Prompt: [Eiffel Tower] ───> (Early Layers: Factual Memory Active) ───> "Paris"
Visual Prompt: [Raw Pixels] ───> (Early Layers: Resolving Identity) ───> (Deep Layers: Identity Found) ───> Too Late! Memory Circuits Inactive
Venhoff et al. discovered that visual entity recognition occurs far too late in the internal processing layers of the language backbone. By the time the network has successfully figured out the identity of the visual object, it has already passed the narrow window of layers where its factual memory circuits are active. The authors refer to this as being "too late to recall."
This can also be understood as a misuse of the model's "computational depth." Cohen 0002 et al. demonstrate that VLMs spend too much of their layer capacity simply decoding the subject from the image. By the time the model shifts from perception to reasoning, it has insufficient layers remaining to retrieve and process the facts it actually knows, leading to an operational performance drop.
The Failure of the Vision-Language Bridge
This layer-wise misalignment is compounded by how vision-language bridges (like linear projectors or perceiver resamplers) are trained.
Conventional pretraining objectives, such as image captioning, optimize for coarse visual descriptions rather than precise factual querying. As Ashok et al. argue, these objectives do not adequately train the bridge to transform visual features into the exact symbolic representations required to activate downstream factual memory.
Furthermore, fine-grained visual details are often lost or diluted when raw pixels are compressed into a small number of visual tokens compatible with the language model backbone. Kim notes that while a model can easily classify an object when given an expert-level text description, it fails when given the image because the critical visual nuances are washed out during feature mapping. This creates what Zhao et al. term "cognitive misalignment"—a state where the vision encoder successfully captures a feature, but the language backbone fails to map it to its internal knowledge base.
This bottleneck extends beyond static encyclopedic facts to episodic and personalized data. For example, Oh et al. show that even when a VLM can visually match a query image to a user's history, it struggles to retrieve the specific textual metadata or context associated with that match, failing to ground its interactions in visual evidence.
Bridging the Gap
To overcome this bottleneck, researchers are shifting away from simply scaling up models and are instead focusing on explicit alignment strategies.
One approach is to enforce multi-granularity supervision during training. For instance, the Entity-Enhanced Cognitive Alignment (EECA) framework introduced by Zhao et al. utilizes high-resolution visual branches and hierarchical entity supervision to ensure that the visual representations directly evoke the correct linguistic concepts. Another strategy, proposed by Kim, forces models to explicitly identify distinctive visual attributes before attempting fine-grained classification, effectively restructuring the model's internal computational path.
Additionally, researchers are exploring automated ways to map out these discrepancies. Sehgal et al. developed ESCHER, a framework that uses VLMs as critics to discover where textual knowledge and visual nuances diverge, automatically building visual concept libraries to fix the areas where the language model's expectations mismatch reality.
Ultimately, solving the cross-modal grounding gap requires recognizing that seeing an object and knowing a fact about it are handled by fundamentally different mechanisms within a transformer. True multimodal intelligence requires aligning these processes not just in the dataset, but across the internal computational layers of the model itself.
Key Foundational Work
Can VLMs Recall Factual Associations From Visual References? (2025) — contributed 4 concept elements to this direction.
Preface (2025) — contributed 3 concept elements to this direction.
Finer: Investigating and Enhancing Fine-Grained Visual Concept Recognition in Large Vision Language Models (2024) — contributed 2 concept elements to this direction.