Abstract
Spatial reasoning in vision language models (VLMs) remains fragile when semantics hinge on subtle temporal or geometric cues. We introduce a synthetic benchmark that probes two complementary skills: situational awareness (recognizing whether an interaction is harmful or benign) and spatial awareness (tracking who does what to whom, and reasoning about relative positions and motion). Through minimal video pairs, we test three challenges: distinguishing violence from benign activity, binding assailant roles across viewpoints, and judging fine-grained trajectory alignment. While we evaluate recent VLMs in a training-free setting, the benchmark is applicable to any video classification model. Results show performance only slightly above chance across tasks. A simple aid, stable color cues, partly reduces assailant role confusions but does not resolve the underlying weakness. By releasing data and code, we aim to provide reproducible diagnostics and seed exploration of lightweight spatial priors to complement large-scale pretraining.
Introduction
While Vision-Language Models (VLMs) like GPT-4o and Qwen2.5-VL demonstrate impressive capabilities in general scene description and captioning, they possess a significant "blind spot" regarding basic spatial and situational context. Researchers from Delft University of Technology have identified that these models often perform near chance when the meaning of a video depends on subtle geometric or temporal cues. For instance, a model might correctly identify "two people moving" but fail to distinguish between a fight and a dance, or fail to identify which individual is the "attacker" versus the "victim."
This research addresses two specific gaps in current VLM capabilities:
- Situational Awareness: The ability to recognize the nature of an interaction (e.g., harmful vs. benign) when the visual context and characters remain constant.
- Spatial Awareness: The ability to track "who does what to whom" and reason about relative positions and motion trajectories.
Methodology and Evaluation Pipeline
To expose these weaknesses, the researchers developed a compact, highly controlled synthetic benchmark using Unreal Engine 5.5. This environment allows for a "minimal-pair" design, where two videos are generated to be identical in every respect except for a single controlled variable, such as which character initiates a physical altercation.
The evaluation employs a training-free pipeline. Because VLMs typically provide free-form text answers, their responses are passed through a secondary text classifier (BART). This classifier maps the model's explanation to a specific label (e.g., "Anomaly" vs. "Normal"), ensuring a standardized metric for comparison across different model architectures.
The Synthetic Benchmark
The benchmark, hosted on Hugging Face, consists of 40 videos per experiment class across five distinct scenes with randomized characters. The tasks are categorized into three primary challenges:
- Situational Discrimination: Distinguishing between fighting and benign activities like dancing or standing idle.
- Role Binding: Identifying the attacker versus the victim from varying camera viewpoints.
- Following: Determining if one character is intentionally following another based on a subtle 30° trajectory alignment.
Key Insights and Results
The evaluation of four state-of-the-art models (Gemma3-4B, NVILA-8B, Qwen2.5-VL-7B, and VideoLLaMA3-7B) revealed several critical findings:
The "Left/Right" Confusion
Models struggle immensely with viewpoint-relative terms. A model might correctly identify an attacker when viewed from one angle but fail when the camera moves to the opposite side, even if the action itself is identical. This suggests a fundamental weakness in anchoring motion to relative spatial terms.
Color as a Linguistic Anchor
Interestingly, when characters were given stable color identifiers (e.g., "the person in red"), model performance improved significantly. This indicates that the failure is not necessarily in tracking the motion itself, but in the linguistic mapping of that motion to spatial roles.
Performance Metrics
- Following Task: Most models hovered around 50% accuracy, essentially equivalent to random guessing. They frequently confused "walking in the same direction" with intentional "following."
- Situational Discrimination: Only NVILA and VideoLLaMA3 could reliably separate violent anomalies from benign dancing. Other models either "over-flagged" everything as an anomaly or were overly optimistic, labeling most interactions as benign.
Conclusion and Future Directions
This work serves as a diagnostic instrument, highlighting that current VLMs lack the explicit spatial grounding necessary for safety-critical applications. While the current benchmark is limited to two actors and short clips, the researchers plan to expand the dataset to include multi-party interactions, occlusion tests, and egocentric viewpoints. Until these spatial reasoning gaps are closed, VLMs cannot be fully trusted for autonomous monitoring or collaborative robotics.
Summary updated