MOSU: Autonomous Long-range Robot Navigation with Multi-modal Scene Understanding

Jing Liang, Kasun Weerakoon, Daeun Song, Senthurbavan Kirubaharan, Xuesu Xiao, Dinesh ManochaarXiv 2025 · DOI · arXiv

Abstract

Autonomous long-range robot navigation in complex outdoor environments requires a balance between global goal-seeking and local semantic and social awareness. We present MOSU (Multi-modal Scene Understanding), a system that decomposes navigation into high-level routing, multi-modal trajectory generation, and motion planning. By integrating LiDAR geometry, RGB semantic segmentation, and Vision-Language Models (VLMs), MOSU achieves a 77% traversability score on the Global Navigation Dataset (GND), outperforming state-of-the-art baselines by 10% while maintaining social compliance in human-centric spaces.

Introduction

Navigating a robot across a college campus or urban environment involves more than simple obstacle avoidance; it requires an understanding of environmental context. Traditional navigation systems often rely on expensive, high-definition maps or struggle with semantic ambiguity—for instance, failing to distinguish between traversable sidewalks and non-traversable grass that appears geometrically similar to LiDAR. Furthermore, hard-coding social norms, such as slowing down for pedestrians or staying within crosswalks, remains a significant challenge for traditional algorithms. MOSU addresses these issues by bridging the gap between high-level global routing and low-level contextual behavior.

System Architecture and Methodology

MOSU decomposes the navigation problem into three distinct stages to ensure scalability and robustness.

MOSU System Architecture: The pipeline flows from sensor data through routing and multi-modal trajectory selection to final motion planning.

1. Routing

Instead of complex global maps, MOSU utilizes standard GPS and QGIS to set subgoals approximately 50 meters apart. This provides a directional guide without requiring centimeter-level map accuracy, making the system easier to deploy in new environments.

2. Trajectory Generation and Selection

This core component evaluates candidate paths using three distinct modalities:

  • Geometric (CVAE): A Conditional Variational Autoencoder processes LiDAR data to identify physically clear paths.
  • Semantic (Mask2Former): RGB images are segmented to distinguish traversable surfaces (roads, sidewalks) from non-traversable ones (vegetation, buildings).
  • Contextual (VLM): A Vision-Language Model (Gemini) ranks paths based on social cues and natural language instructions, such as "stay in crosswalks" or "keep away from groups of pedestrians."

3. Motion Planning

The final trajectory is selected via a weighted sum of scores from these three modalities. A Dynamic Window Approach (DWA) translates the selection into motor commands. Notably, the VLM can dynamically adjust the robot's speed between "Normal" and "Slow" modes based on real-time crowd density.

Experimental Setup and Datasets

The researchers evaluated MOSU using the Global Navigation Dataset (GND), which encompasses 10 different university campuses featuring diverse urban and rural environments.

The GND dataset includes challenging real-world scenarios such as narrow spaces, off-road terrain, crosswalks, and social areas.

The dataset presents complex challenges, including narrow passages and busy social areas. MOSU was benchmarked against several state-of-the-art models, including MTG, DTG, ViNT, and PIVOT.

Key Results and Insights

  • Superior Traversability: MOSU achieved a 77% traversability score, a 10% improvement over the next best model (PIVOT at 70%) and significantly higher than geometry-only models like MTG (61%).
  • Navigation Efficiency: The system maintained a 73% "Distance to Target" score, demonstrating that it reaches goals efficiently rather than just wandering safely.
  • Multi-Modal Synergy: The study found that geometry and semantics are complementary; LiDAR provides 3D precision for maneuvering, while semantic segmentation identifies appropriate surfaces that LiDAR might misinterpret.
  • Social Reasoning: By leveraging VLMs, the robot can apply "common sense" to follow social norms that are difficult to capture with traditional cost-maps.
  • Real-Time Performance: Despite the computational load of a VLM, the system maintains an inference time of approximately 2.3 seconds, making it feasible for real-world deployment.

Limitations and Future Work

While MOSU represents a significant step forward, some challenges remain. The system occasionally struggles with "vertical hazards" like small cliffs or ramp edges. Additionally, like many learning-based models, it can produce noisy paths in out-of-distribution (OOD) scenes radically different from its training data. Future work will focus on incorporating robust foundation geometric models to better handle 3D constraints and improving terrain judgment for features like ramps.

Summary updated

Abstract

We present MOSU, a novel autonomous long-range navigation system that enhances global navigation for mobile robots through multimodal perception and on-road scene understanding. MOSU addresses the outdoor robot navigation challenge by integrating geometric, semantic, and contextual information to ensure comprehensive scene understanding. The system combines GPS and QGIS map-based routing for high-level global path planning and multi-modal trajectory generation for local navigation refinement. For trajectory generation, MOSU leverages multi-modalities: LiDAR-based geometric data for precise obstacle avoidance, image-based semantic segmentation for traversability assessment, and Vision-Language Models (VLMs) to capture social context and enable the robot to adhere to social norms in complex environments. This multi-modal integration improves scene understanding and enhances traversability, allowing the robot to adapt to diverse outdoor conditions. We evaluate our system in real-world on-road environments and benchmark it on the GND dataset, achieving a 10% improvement in traversability on navigable terrains while maintaining a comparable navigation distance to existing global navigation methods.
BibTeX
@misc{liang2025mosu,
  author = {Jing Liang and Kasun Weerakoon and Daeun Song and Senthurbavan Kirubaharan and Xuesu Xiao and Dinesh Manocha},
  title = {{MOSU: Autonomous Long-range Robot Navigation with Multi-modal Scene Understanding}},
  year = {2025},
  eprint = {2507.04686},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2507.04686},
}
Figure