TL;DR
MOSU2 is a long-range navigation system that enables robots to handle complex social and traffic scenarios by grounding Vision-Language Models (VLMs) in metric geometric and dynamic data. By using a two-stage reasoning architecture, the system selectively triggers deep deliberation only when needed, boosting navigation success from 61.8% to 94.2% in challenging real-world environments.
What Problem Does This Paper Address?
Long-range outdoor navigation requires more than just avoiding static obstacles; robots must navigate "socially" by yielding to pedestrians, interpreting traffic signals, and understanding temporary restrictions like construction zones.
While Vision-Language Models (VLMs) are excellent at interpreting high-level context (e.g., "that person is waving me to stop"), they typically reason over single images that lack metric geometry and motion. A robot can't safely decide to cross a street based on a single snapshot; it needs to know the velocity of an approaching car. Furthermore, running a full VLM analysis at every control step is computationally prohibitive for real-time robotics. MOSU2 addresses this "context-metric gap" and the latency bottleneck.
Methods
MOSU2 builds upon the original MOSU framework but introduces a novel two-stage reasoning architecture and a VLM-retrievable local map.
- Global Routing & Trajectory Generation: The system uses GPS for coarse guidance and a Conditional Variational Autoencoder (CVAE) to generate a set of candidate trajectories based on LiDAR and robot velocity.
- Multi-modal Traversability: Candidate paths are filtered using both geometric data (LiDAR) and semantic segmentation (using SAM3) to ensure they stay on robot-appropriate terrain (e.g., sidewalks for wheeled robots).
- VLM-retrievable Local Map: This is a key innovation. The system constructs a 2D local map that associates semantic labels (from SAM3) with metric positions (from LiDAR) and estimated velocities.
- Two-Stage Reasoning:
- Stage 1 (Screening): A VLM quickly examines a single image to see if the situation is routine. If it detects a potential interaction (like a pedestrian or a traffic light), it requests specific metric evidence.
- Stage 2 (Deliberation): Triggered only when needed, this stage analyzes a sequence of recent images and retrieves specific metric data (e.g., "how fast is the car at coordinates X, Y moving?") from the local map to make a final navigation decision.
Datasets
- GND Dataset: Used for quantitative evaluation of traversability and distance-to-target metrics. It contains observations from ten different university campuses.
- Real-World Trials: The authors conducted extensive physical experiments using three different robot platforms (Husky, Unitree Go2, and Diablo) across five scenario groups: human gestures, construction sites, signalized crosswalks, shared roads, and sidewalk baselines.
Key Insights and Ideas
- Interaction Detection vs. Interpretation: The authors argue that a single image is often enough to detect that an interaction is happening, but interpreting that interaction requires temporal context (image sequences) and metric grounding (velocities).
- Selective Deliberation: By using the first stage as a "gatekeeper," the system avoids the high latency of complex VLM reasoning during routine navigation, reserving compute for ambiguous or high-stakes situations.
- Grounding Qualitative Cues: MOSU2 allows a VLM to "ask" the map for quantitative data. Instead of just seeing a "car," the VLM can reason about a "car moving at 2 m/s," allowing for much more reliable yielding behavior.
Key Results
- Success Rate: MOSU2 achieved a mean navigation success rate of 94.2%, a massive jump from the 61.8% achieved by its predecessor, MOSU.
- Condition Detection: The accuracy of detecting specific social/traffic conditions (like a "stop" gesture or a red light) improved from 72.2% to 91.4%.
- Zero-Shot Reliability: In construction site scenarios, MOSU2 reached 100% success, doubling the performance of the previous version by correctly grounding visual signs in the local geometric map.
- Latency Trade-off: The improved reliability comes at a cost; the mean decision latency increased by about 2.2x (from ~1.7s to ~3.8s) when the second stage was invoked.
Limitations and Future Work
- Perception Sensitivity: The system is highly dependent on the quality of LiDAR returns and segmentation. Sparse LiDAR or "out-of-distribution" visual scenes can lead to mapping errors.
- The "Missed Cue" Risk: If the first-stage screening fails to identify a subtle social cue, the second deliberative stage is never triggered, potentially leading to unsafe behavior.
- Latency: While the two-stage approach helps, a ~5-second latency in complex scenes is still high for fast-moving environments. Future work aims to explore lightweight, on-device screening models and more robust temporal association for motion estimation.
Summary updated