TL;DR
Adding more "helpful" information to Vision-Language Models (VLMs) often backfires. This study reveals that while targeted spatial cues can boost performance, piling on multiple contexts, loosely related commonsense knowledge, or Chain-of-Thought (CoT) instructions frequently leads to "cognitive overload" and degraded reasoning.
What Problem Does This Paper Address?
Vision-Language Models (VLMs) like GPT-4V or LLaVA are impressively capable, yet they consistently stumble on Visual Spatial Reasoning (VSR)—tasks requiring an understanding of "left vs. right," "behind," or "overlapping." To fix this, researchers often "inject" extra information into the prompt at inference time, such as object coordinates, depth data, or commonsense facts (e.g., "dogs usually face the things they are chasing").
However, there is no "free lunch." The authors identify a gap in our understanding: we don't actually know if more information is always better. Does a model get smarter with ten spatial cues, or does it just get confused? This paper systematically investigates the "sweet spot" for information injection to prevent models from being overwhelmed by noise.
Methods
The researchers treated three representative VLMs (Qwen-2-VL, LLaVA-Next-1.6, and BLIP-3) as black-box systems. Instead of retraining them, they performed controlled input interventions by varying three types of injected data:
- Spatial Contexts (SC): They used off-the-shelf tools (like Grounding DINO and Depth Anything v2) to generate bounding boxes, orientation angles, and depth values. Crucially, they tested both raw numerical values and verbalized descriptions (e.g., "Object A is far").
- Commonsense Knowledge (CK): They retrieved relational triplets from the ATOMIC-COMET knowledge base, varying the "relevance threshold" to see how much loosely related info a model could handle.
- Spatial Reasoning (SR) Instructions: They applied Chain-of-Thought (CoT) prompting, forcing the model to explain its spatial logic step-by-step.
Datasets
The study evaluated models on two public benchmarks:
- VSR (Visual Spatial Reasoning): Focuses on fine-grained spatial relations across 66 categories (e.g., "under," "touching").
- EmbSpatial: Focuses on reasoning within cluttered, object-dense scenes typical of embodied AI/robotics environments.
Key Insights and Ideas
- The "Cognitive Overload" Effect: Performance generally peaks with just one well-chosen spatial context. As more contexts are appended, accuracy declines. By the time five or more contexts are added, the models often perform worse than they did with a single cue.
- Verbalization > Numbers: VLMs struggle to integrate raw coordinates or metric depth values. They perform significantly better when these numbers are translated into natural language (e.g., "the cat is on the left"). Raw numeric data often acts as "noise" that the model cannot effectively normalize.
- The CoT Double-Edged Sword: Chain-of-Thought reasoning is only helpful when the model's initial "grounding" (its basic perception of where things are) is accurate. If the frame of reference is ambiguous (e.g., "to the right" from the camera's view vs. the object's view), CoT actually amplifies errors by forcing the model to build a logical-sounding path based on a false premise.
- Commonsense Needs a Filter: Injecting commonsense only helps at very high relevance thresholds (0.7–0.9 similarity). Lowering the bar to include more "maybe relevant" facts quickly degrades performance.
Key Results
- Accuracy Gains: Targeted spatial context injection improved accuracy by up to 9.6%.
- CoT Divergence: On the EmbSpatial dataset, CoT improved performance for LLaVA and BLIP-3 by up to 12.2%. However, on the VSR dataset, it actually degraded accuracy by up to 2.0% due to grounding ambiguity.
- Diminishing Returns: In almost all tests, the "full combination" (SC + CK + SR) performed worse than simpler, more selective interventions. This confirms that current VLMs lack the "integration capacity" to handle high-density multi-modal prompts.
Limitations and Future Work
The authors acknowledge that their study treats VLMs as fixed black boxes. It remains to be seen if models specifically trained on high-density spatial data would exhibit the same "overload" behavior. Additionally, the quality of the injected info depends on the "off-the-shelf" perception models used (like DINO); if the perception tool makes a mistake, the VLM is almost guaranteed to follow suit.
Future work could focus on dynamic information selection: building a "meta-module" that decides exactly which single piece of information is most useful for a specific image-query pair, rather than using a static injection pipeline.
Summary updated