The Spatial Perception Gap in Medical Vision Language Models

22 concepts13 papers2 proposals

Large Vision-Language Models (LVLMs) have demonstrated remarkable capabilities in general visual understanding, but when deployed in the medical domain, they encounter a striking paradox. A model can look at a complex radiological scan and correctly identify a rare pathology, yet utterly fail to determine whether that pathology is on the left or right side of the patient's body.

This disconnect between semantic recognition and geometric awareness is known as the perception-reasoning gap. While medical generalist models excel at high-level classification and pattern recognition, they frequently suffer from weak spatial perception, coordinate hallucination, and an inability to navigate the 3D anatomical relationships crucial for clinical diagnostics.

The Illusion of Competence: Recognition vs. Localization

To understand why this gap exists, we must look at how standard vision-language models process images. Most architectures are trained on holistic image-text pairs using contrastive or generative objectives. This forces the visual encoder to capture global features and textures associated with specific diseases. For instance, a model learns that a certain hazy texture in a chest X-ray corresponds to "pneumonia."

However, as Zhou et al. demonstrate, identifying an anatomical structure does not mean the model understands its spatial orientation or relative position. In clinical practice, a diagnosis rarely depends on a single isolated finding; it relies on understanding the precise spatial arrangement, proximity, and boundaries of biological features.

When evaluated on granular spatial tasks, the performance of state-of-the-art generalist models drops sharply:

  • Lateralization Errors: Models frequently confuse the left and right orientations of the human body. Wang et al. highlight that even when a model correctly identifies a pathological feature, it often misreports its lateral anatomical location.
  • Keypoint and Boundary Failures: Comprehensive benchmarks on noisy modalities like ultrasound reveal that models struggle to maintain a consistent internal mental map of anatomical structures. Comprehensive evaluations by Le et al. and Le et al. show a massive performance discrepancy between image-level disease classification and precise spatial tasks like keypoint detection or lesion localization.
  • The Memorization Shortcut: Why do models appear spatially aware in some evaluations? Wolf et al. discovered that generalist models often ignore the actual image pixels entirely. Instead, they rely on "memorized" anatomical knowledge from their text-based pre-training data. If a specific tumor most commonly occurs in a certain organ quadrant, the model will guess that location by default, failing dangerously when presented with atypical or rare spatial presentations.

Without precise spatial grounding, the clinical utility of automated report generation is severely limited. As noted by Ranjit et al. and Zhu et al., a model that cannot reliably pinpoint the location of a lesion or a medical instrument cannot be trusted for complex clinical decision-making or surgical planning.

[Global Image-Text Matching] ---> Identifies "What" (e.g., Tumor)
                                      |
                                      v (The Spatial Gap)
[Pixel-Level Grounding]      ---> Fails at "Where" (e.g., Left vs. Right, Coordinates)

Why General Visual Encoders Fail Medical Geometry

In natural images, spatial relationships are often explicit and highly varied (e.g., "a cat sitting on top of a mat"). Visual encoders like CLIP learn these relationships through diverse web-scale data.

Medical images, however, operate under rigid, highly constrained anatomical geometries. A CT scan or an ultrasound slice presents a dense, grayscale, and often noisy cross-section where boundaries are subtle and structures overlap. General-purpose visual encoders are simply not optimized for these specific spatial constraints (Wolf et al.).

Furthermore, standard autoregressive language generation does not inherently penalize spatial inaccuracies. If a model generates a text report containing coordinate hallucinations, the language loss treats it similarly to any other token mismatch, failing to capture the severe geometric penalty of a misplaced bounding box or an incorrect directional orientation (Yan et al.).

Bridging the Gap: Architectural Solutions

To overcome this limitation, recent research focuses on explicitly binding high-level semantic reasoning to low-level pixel geometry. Rather than treating spatial perception as an implicit byproduct of text generation, these approaches introduce structural, region-centric, and agentic constraints.

1. Unifying Text and Pixel-Level Segmentation

One of the most effective ways to force a model to respect spatial boundaries is to interleave text generation with explicit pixel segmentation.

The MedSIGHT framework by Chang et al. unifies visual comprehension and pixel-level segmentation within a single model. By forcing the model's internal representations to simultaneously predict text tokens and segment masks, the language output remains anchored to actual physical coordinates.

Similarly, the RAU framework proposed by Li et al. pairs the high-level reasoning of an LVLM with the precise localization power of specialized segmentation models like SAM2, using a single annotated reference image to map complex anatomical boundaries across entirely different imaging modalities.

[Input Scan] ---> [Unified Med-LVLM] ---> Text: "Tumor detected in the left lobe."
                                     ---> Mask:  [Pixel-level Segmentation Map]
                                                    (Forces geometric alignment)

2. Explicit Anatomical and Knowledge Alignment

Instead of relying on raw image-text matching, models can be forced to learn the underlying structural map of the human body.

The authors of Anatomy-VLM address this by designing a model that explicitly detects, localizes, and reasons about individual anatomical regions in a structured manner, mimicking the systematic workflow of a human radiologist.

To combat positional biases and coordinate hallucinations, Yan et al. introduce MMCAP, a framework that injects external structured medical knowledge graphs directly into the visual encoding process. This explicitly aligns visual features with known anatomical hierarchies and directional orientations, preventing the model from confusing relative positions when multiple pathologies co-occur.

3. Abnormal-Aware Feedback Loops

Another strategy involves modifying the training objective to penalize spatial blindness. Zhou et al. introduce UMed-LVLM, which utilizes a two-stage training process with reinforcement learning and "abnormal-aware" feedback. This framework explicitly rewards the model for precise localization and grounding, ensuring that text rationales are directly tied to verified regions of interest, as also explored by Ma et al..

4. Agentic Tool Orchestration

If a single generalist neural network cannot master both abstract language reasoning and precise pixel geometry, an alternative is to split the labor.

Li et al. developed MMedAgent, a multi-modal agent that acts as an "action planner." Instead of forcing the central LLM to calculate coordinates or draw bounding boxes directly—tasks where it is prone to hallucination—the agent orchestrates a suite of specialized, pre-trained expert tools for grounding, segmentation, and clinical report generation. This modular approach ensures that high-level clinical reasoning is always backed by mathematically precise spatial tools.

Key Foundational Work

Beyond the Hype: A dispassionate look at vision-language models in medical scenario (2024) — contributed 3 concept elements to this direction.
Multi-modal Concept Alignment Pre-training for Generative Medical Visual Question Answering (2024) — contributed 2 concept elements to this direction.
MMedAgent: Learning to Use Medical Tools with Multi-modal Agent (2024) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_weak_spatial_perception_b67b31b0,
  title = {weak spatial perception},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/weak-spatial-perception/rdr_b67b31b06c41473aa66f7e8cf9a32826},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_weak_spatial_perception_b67b31b0_v1,
  title = {weak spatial perception},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/weak-spatial-perception/rdr_b67b31b06c41473aa66f7e8cf9a32826/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake