SocialNav-SUB: Benchmarking VLMs for Scene Understanding in Social Robot Navigation

Michael J. Munje, Chen Tang, Shuijing Liu, Zichao Hu, Yifeng Zhu, Jiaxun Cui, Garrett Warnell, Joydeep Biswas, Peter Stone

For a robot to navigate a crowded hallway or a busy campus, it isn't enough to simply avoid obstacles. It must be socially compliant—understanding when to yield to a hurried pedestrian or how to follow a group's flow. This requires high-level scene understanding across three critical dimensions: spatial reasoning (locating individuals), spatiotemporal reasoning (predicting movement), and social reasoning (inferring intentions).

While Large Vision-Language Models (VLMs) like GPT-4o or Gemini have shown impressive general-purpose reasoning, their ability to handle the high-stakes nuances of social navigation remains unproven. This research introduces SocialNav-SUB, the first comprehensive Visual Question Answering (VQA) benchmark designed to determine if VLMs are ready to serve as the "brains" of social robots.

The SocialNav-SUB Benchmark

The researchers developed a pipeline to transform real-world robot navigation data into a rigorous VQA challenge. Using the SCAND dataset—recorded from teleoperated robots on a university campus—they applied the PHALP tracking algorithm to identify pedestrians in 3D space.

Overview of the SocialNav-SUB benchmark creation pipeline, dataset statistics, and the core research topics for scene understanding.

To assist the models, the benchmark provides both a front-view image sequence and a Bird’s-Eye View (BEV) map. Pedestrians are color-coded and numbered in both views to ensure clear references. The resulting dataset consists of 4,968 unique multiple-choice questions across 60 challenging scenarios. Because social navigation can be subjective, performance is measured using the Probability of Agreement (PA) and Consensus-Weighted PA (CWPA), comparing VLM answers to a distribution of responses from 153 human participants.

Experimental Results

The evaluation of state-of-the-art VLMs revealed a surprising gap between semantic understanding and physical grounding. While models showed promise in social reasoning, they frequently struggled with basic spatial tasks.

Quantitative performance comparison of various VLMs against human baselines and rule-based systems across reasoning categories.

Key Insights:

  • The Spatial Reasoning Bottleneck: VLMs are relatively better at interpreting social cues (e.g., "Is this person avoiding the robot?") than basic spatial tasks (e.g., "Is the person to the left or right?"). This suggests that while VLMs have "read" about social norms, they struggle to map those concepts to 3D physical coordinates.
  • Rule-Based Systems Still Lead: A simple baseline using hand-crafted geometric rules outperformed most VLMs. This highlights that current foundation models are not yet a "drop-in" replacement for traditional robotics stacks.
  • Top Performer: OpenAI’s o4-mini achieved the highest overall VLM score with a 62% PA, yet it still trailed behind the Rule-Based baseline (64%) and the Human Oracle (74%).
  • Open-Source Performance: LLaVa-Next-Video, an open-source model capable of running locally, struggled significantly in spatial tasks (35% PA), indicating a long road ahead for on-device deployment.

Discussion and Future Work

The results confirm that "understanding the scene" is the primary hurdle for VLM-based navigation. While Chain-of-Thought (CoT) prompting significantly improves social reasoning by forcing models to synthesize spatial data, current state-of-the-art VLMs remain too computationally expensive for real-time robot control.

Future research will likely focus on distillation—shrinking the reasoning capabilities of massive models into smaller, faster versions that can run on onboard hardware without significant latency. Additionally, the authors suggest that "hybrid" systems, combining dedicated perception modules for spatial grounding with VLM-based social reasoning, may offer the most viable path toward truly social robots.

Summary updated

BibTeX
@inproceedings{munje2025socialnav,
  author = {Michael J. Munje and Chen Tang and Shuijing Liu and Zichao Hu and Yifeng Zhu and Jiaxun Cui and Garrett Warnell and Joydeep Biswas and Peter Stone},
  title = {{SocialNav-SUB: Benchmarking VLMs for Scene Understanding in Social Robot Navigation}},
  booktitle = {Conference on Robot Learning},
  year = {2025},
  url = {https://openreview.net/forum?id=xVDj9uq6K3},
}
Figure