When More Is Less: A Systematic Analysis of Spatial and Commonsense Information for Visual Spatial Reasoning

Muku Akasaka, Soyeon Caren Han

TL;DR

Adding more "helpful" information to Vision-Language Models (VLMs) often backfires. This study reveals that while targeted spatial cues can boost performance, piling on multiple contexts, loosely related commonsense knowledge, or Chain-of-Thought (CoT) instructions frequently leads to "cognitive overload" and degraded reasoning.

What Problem Does This Paper Address?

Vision-Language Models (VLMs) like GPT-4V or LLaVA are impressively capable, yet they consistently stumble on Visual Spatial Reasoning (VSR)—tasks requiring an understanding of "left vs. right," "behind," or "overlapping." To fix this, researchers often "inject" extra information into the prompt at inference time, such as object coordinates, depth data, or commonsense facts (e.g., "dogs usually face the things they are chasing").

However, there is no "free lunch." The authors identify a gap in our understanding: we don't actually know if more information is always better. Does a model get smarter with ten spatial cues, or does it just get confused? This paper systematically investigates the "sweet spot" for information injection to prevent models from being overwhelmed by noise.

Methods

The researchers treated three representative VLMs (Qwen-2-VL, LLaVA-Next-1.6, and BLIP-3) as black-box systems. Instead of retraining them, they performed controlled input interventions by varying three types of injected data:

  1. Spatial Contexts (SC): They used off-the-shelf tools (like Grounding DINO and Depth Anything v2) to generate bounding boxes, orientation angles, and depth values. Crucially, they tested both raw numerical values and verbalized descriptions (e.g., "Object A is far").
  2. Commonsense Knowledge (CK): They retrieved relational triplets from the ATOMIC-COMET knowledge base, varying the "relevance threshold" to see how much loosely related info a model could handle.
  3. Spatial Reasoning (SR) Instructions: They applied Chain-of-Thought (CoT) prompting, forcing the model to explain its spatial logic step-by-step.

Datasets

The study evaluated models on two public benchmarks:

  • VSR (Visual Spatial Reasoning): Focuses on fine-grained spatial relations across 66 categories (e.g., "under," "touching").
  • EmbSpatial: Focuses on reasoning within cluttered, object-dense scenes typical of embodied AI/robotics environments.

Key Insights and Ideas

  • The "Cognitive Overload" Effect: Performance generally peaks with just one well-chosen spatial context. As more contexts are appended, accuracy declines. By the time five or more contexts are added, the models often perform worse than they did with a single cue.
  • Verbalization > Numbers: VLMs struggle to integrate raw coordinates or metric depth values. They perform significantly better when these numbers are translated into natural language (e.g., "the cat is on the left"). Raw numeric data often acts as "noise" that the model cannot effectively normalize.
  • The CoT Double-Edged Sword: Chain-of-Thought reasoning is only helpful when the model's initial "grounding" (its basic perception of where things are) is accurate. If the frame of reference is ambiguous (e.g., "to the right" from the camera's view vs. the object's view), CoT actually amplifies errors by forcing the model to build a logical-sounding path based on a false premise.
  • Commonsense Needs a Filter: Injecting commonsense only helps at very high relevance thresholds (0.7–0.9 similarity). Lowering the bar to include more "maybe relevant" facts quickly degrades performance.

Key Results

  • Accuracy Gains: Targeted spatial context injection improved accuracy by up to 9.6%.
  • CoT Divergence: On the EmbSpatial dataset, CoT improved performance for LLaVA and BLIP-3 by up to 12.2%. However, on the VSR dataset, it actually degraded accuracy by up to 2.0% due to grounding ambiguity.
  • Diminishing Returns: In almost all tests, the "full combination" (SC + CK + SR) performed worse than simpler, more selective interventions. This confirms that current VLMs lack the "integration capacity" to handle high-density multi-modal prompts.

Limitations and Future Work

The authors acknowledge that their study treats VLMs as fixed black boxes. It remains to be seen if models specifically trained on high-density spatial data would exhibit the same "overload" behavior. Additionally, the quality of the injected info depends on the "off-the-shelf" perception models used (like DINO); if the perception tool makes a mistake, the VLM is almost guaranteed to follow suit.

Future work could focus on dynamic information selection: building a "meta-module" that decides exactly which single piece of information is most useful for a specific image-query pair, rather than using a static injection pipeline.

Summary updated

Abstract

Visual spatial reasoning (VSR) remains challenging for modern vision-language models (VLMs), despite advances in multimodal architectures. A common strategy is to inject additional information at inference time, such as explicit spatial cues, external commonsense knowledge, or chain-of-thought (CoT) reasoning instructions. However, it remains unclear when such information genuinely improves reasoning and when it introduces noise. In this paper, we conduct a hypothesis-driven analysis of information injection for VSR across three representative VLMs and two public benchmarks. We examine (i) the type and number of spatial contexts, (ii) the amount and relevance of injected commonsense knowledge, and (iii) the interaction between spatial grounding and CoT prompting. Our results reveal a consistent pattern: more information does not necessarily yield better reasoning. Targeted single spatial cues outperform multi-context aggregation, excessive or weakly relevant commonsense knowledge degrades performance, and CoT prompting improves accuracy only when spatial grounding is sufficiently precise. These findings highlight the importance of selective, task-aligned information injection and provide practical guidance for designing reliable multimodal reasoning pipelines.
BibTeX
@misc{akasaka2026when,
  author = {Muku Akasaka and Soyeon Caren Han},
  title = {{When More Is Less: A Systematic Analysis of Spatial and Commonsense Information for Visual Spatial Reasoning}},
  year = {2026},
  eprint = {2602.21619},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2602.21619},
}
Figure