MOSU2: Long-range Navigation System with Multi-modal Scene Understanding

Jing Liang, Zhenyu Hou, Senthurbavan Kirubaharan, Md Safwan Mondal, Ashok Gudivada, Xuesu Xiao, Dinesh ManochaarXiv 2026 · arXiv

TL;DR

MOSU2 is a long-range navigation system that enables robots to handle complex social and traffic scenarios by grounding Vision-Language Models (VLMs) in metric geometric and dynamic data. By using a two-stage reasoning architecture, the system selectively triggers deep deliberation only when needed, boosting navigation success from 61.8% to 94.2% in challenging real-world environments.

What Problem Does This Paper Address?

Long-range outdoor navigation requires more than just avoiding static obstacles; robots must navigate "socially" by yielding to pedestrians, interpreting traffic signals, and understanding temporary restrictions like construction zones.

While Vision-Language Models (VLMs) are excellent at interpreting high-level context (e.g., "that person is waving me to stop"), they typically reason over single images that lack metric geometry and motion. A robot can't safely decide to cross a street based on a single snapshot; it needs to know the velocity of an approaching car. Furthermore, running a full VLM analysis at every control step is computationally prohibitive for real-time robotics. MOSU2 addresses this "context-metric gap" and the latency bottleneck.

Methods

MOSU2 builds upon the original MOSU framework but introduces a novel two-stage reasoning architecture and a VLM-retrievable local map.

  1. Global Routing & Trajectory Generation: The system uses GPS for coarse guidance and a Conditional Variational Autoencoder (CVAE) to generate a set of candidate trajectories based on LiDAR and robot velocity.
  2. Multi-modal Traversability: Candidate paths are filtered using both geometric data (LiDAR) and semantic segmentation (using SAM3) to ensure they stay on robot-appropriate terrain (e.g., sidewalks for wheeled robots).
  3. VLM-retrievable Local Map: This is a key innovation. The system constructs a 2D local map that associates semantic labels (from SAM3) with metric positions (from LiDAR) and estimated velocities.
  4. Two-Stage Reasoning:
    • Stage 1 (Screening): A VLM quickly examines a single image to see if the situation is routine. If it detects a potential interaction (like a pedestrian or a traffic light), it requests specific metric evidence.
  5. Stage 2 (Deliberation): Triggered only when needed, this stage analyzes a sequence of recent images and retrieves specific metric data (e.g., "how fast is the car at coordinates X, Y moving?") from the local map to make a final navigation decision.
The MOSU2 architecture integrates global routing, local traversability, and two-stage VLM reasoning grounded in a local map.

Datasets

  • GND Dataset: Used for quantitative evaluation of traversability and distance-to-target metrics. It contains observations from ten different university campuses.
  • Real-World Trials: The authors conducted extensive physical experiments using three different robot platforms (Husky, Unitree Go2, and Diablo) across five scenario groups: human gestures, construction sites, signalized crosswalks, shared roads, and sidewalk baselines.

Key Insights and Ideas

  • Interaction Detection vs. Interpretation: The authors argue that a single image is often enough to detect that an interaction is happening, but interpreting that interaction requires temporal context (image sequences) and metric grounding (velocities).
  • Selective Deliberation: By using the first stage as a "gatekeeper," the system avoids the high latency of complex VLM reasoning during routine navigation, reserving compute for ambiguous or high-stakes situations.
  • Grounding Qualitative Cues: MOSU2 allows a VLM to "ask" the map for quantitative data. Instead of just seeing a "car," the VLM can reason about a "car moving at 2 m/s," allowing for much more reliable yielding behavior.
MOSU2 handles diverse challenges including narrow spaces, off-road terrain, crosswalks, and social interactions.

Key Results

  • Success Rate: MOSU2 achieved a mean navigation success rate of 94.2%, a massive jump from the 61.8% achieved by its predecessor, MOSU.
  • Condition Detection: The accuracy of detecting specific social/traffic conditions (like a "stop" gesture or a red light) improved from 72.2% to 91.4%.
  • Zero-Shot Reliability: In construction site scenarios, MOSU2 reached 100% success, doubling the performance of the previous version by correctly grounding visual signs in the local geometric map.
  • Latency Trade-off: The improved reliability comes at a cost; the mean decision latency increased by about 2.2x (from ~1.7s to ~3.8s) when the second stage was invoked.

Limitations and Future Work

  • Perception Sensitivity: The system is highly dependent on the quality of LiDAR returns and segmentation. Sparse LiDAR or "out-of-distribution" visual scenes can lead to mapping errors.
  • The "Missed Cue" Risk: If the first-stage screening fails to identify a subtle social cue, the second deliberative stage is never triggered, potentially leading to unsafe behavior.
  • Latency: While the two-stage approach helps, a ~5-second latency in complex scenes is still high for fast-moving environments. Future work aims to explore lightweight, on-device screening models and more robust temporal association for motion estimation.
Failure cases in simpler models (like confusing grass for sidewalk) highlight the need for the multi-modal approach used in MOSU2.

Summary updated

Abstract

Robots that travel long distances through public outdoor spaces must do more than avoid obstacles: they must remain on terrain suited to their mobility and decide when to proceed, yield, or detour around pedestrians, cyclists, vehicles, traffic signals, and temporary restrictions. These decisions are difficult because their correctness often depends on how a scene evolves, such as whether an approaching car is slowing or a cyclist will cross the robot's path. Vision-language models (VLMs) can interpret such cues, but they reason over images that lack metric geometry and motion. Supplying this evidence at every step is too slow for real-time navigation, and hand-crafted triggers cannot anticipate the open-ended variety of outdoor situations. We present MOSU2, a long-range navigation system with two-stage trajectory selection, in which first-stage contextual reasoning determines when additional evidence is needed and what to retrieve. Only when an interaction remains unresolved does the first stage trigger the second stage to retrieve geometric and dynamic information and align it with scene context to make navigation decisions. Building on MOSU, MOSU2 follows routes from GPS-based route planning and filters candidate trajectories through multi-modal traversability analysis before selecting a trajectory to move the robot. In real-world experiments spanning stop gestures, construction sites, crosswalks, and roads shared with vehicles, cyclists, and pedestrians, MOSU2 raises mean navigation success from 61.8% to 94.2% and condition-detection accuracy from 72.2% to 91.4% over its single-image predecessor, at about 2.2 times its mean decision latency. Project website: https://jinggm.github.io/mosu2/
BibTeX
@misc{liang2026mosu,
  author = {Jing Liang and Zhenyu Hou and Senthurbavan Kirubaharan and Md Safwan Mondal and Ashok Gudivada and Xuesu Xiao and Dinesh Manocha},
  title = {{MOSU2: Long-range Navigation System with Multi-modal Scene Understanding}},
  year = {2026},
  eprint = {2611.11148},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.11148},
}
Figure