Abstract
Autonomous long-range robot navigation in complex outdoor environments requires a balance between global goal-seeking and local semantic and social awareness. We present MOSU (Multi-modal Scene Understanding), a system that decomposes navigation into high-level routing, multi-modal trajectory generation, and motion planning. By integrating LiDAR geometry, RGB semantic segmentation, and Vision-Language Models (VLMs), MOSU achieves a 77% traversability score on the Global Navigation Dataset (GND), outperforming state-of-the-art baselines by 10% while maintaining social compliance in human-centric spaces.
Introduction
Navigating a robot across a college campus or urban environment involves more than simple obstacle avoidance; it requires an understanding of environmental context. Traditional navigation systems often rely on expensive, high-definition maps or struggle with semantic ambiguity—for instance, failing to distinguish between traversable sidewalks and non-traversable grass that appears geometrically similar to LiDAR. Furthermore, hard-coding social norms, such as slowing down for pedestrians or staying within crosswalks, remains a significant challenge for traditional algorithms. MOSU addresses these issues by bridging the gap between high-level global routing and low-level contextual behavior.
System Architecture and Methodology
MOSU decomposes the navigation problem into three distinct stages to ensure scalability and robustness.
1. Routing
Instead of complex global maps, MOSU utilizes standard GPS and QGIS to set subgoals approximately 50 meters apart. This provides a directional guide without requiring centimeter-level map accuracy, making the system easier to deploy in new environments.
2. Trajectory Generation and Selection
This core component evaluates candidate paths using three distinct modalities:
- Geometric (CVAE): A Conditional Variational Autoencoder processes LiDAR data to identify physically clear paths.
- Semantic (Mask2Former): RGB images are segmented to distinguish traversable surfaces (roads, sidewalks) from non-traversable ones (vegetation, buildings).
- Contextual (VLM): A Vision-Language Model (Gemini) ranks paths based on social cues and natural language instructions, such as "stay in crosswalks" or "keep away from groups of pedestrians."
3. Motion Planning
The final trajectory is selected via a weighted sum of scores from these three modalities. A Dynamic Window Approach (DWA) translates the selection into motor commands. Notably, the VLM can dynamically adjust the robot's speed between "Normal" and "Slow" modes based on real-time crowd density.
Experimental Setup and Datasets
The researchers evaluated MOSU using the Global Navigation Dataset (GND), which encompasses 10 different university campuses featuring diverse urban and rural environments.
The dataset presents complex challenges, including narrow passages and busy social areas. MOSU was benchmarked against several state-of-the-art models, including MTG, DTG, ViNT, and PIVOT.
Key Results and Insights
- Superior Traversability: MOSU achieved a 77% traversability score, a 10% improvement over the next best model (PIVOT at 70%) and significantly higher than geometry-only models like MTG (61%).
- Navigation Efficiency: The system maintained a 73% "Distance to Target" score, demonstrating that it reaches goals efficiently rather than just wandering safely.
- Multi-Modal Synergy: The study found that geometry and semantics are complementary; LiDAR provides 3D precision for maneuvering, while semantic segmentation identifies appropriate surfaces that LiDAR might misinterpret.
- Social Reasoning: By leveraging VLMs, the robot can apply "common sense" to follow social norms that are difficult to capture with traditional cost-maps.
- Real-Time Performance: Despite the computational load of a VLM, the system maintains an inference time of approximately 2.3 seconds, making it feasible for real-world deployment.
Limitations and Future Work
While MOSU represents a significant step forward, some challenges remain. The system occasionally struggles with "vertical hazards" like small cliffs or ramp edges. Additionally, like many learning-based models, it can produce noisy paths in out-of-distribution (OOD) scenes radically different from its training data. Future work will focus on incorporating robust foundation geometric models to better handle 3D constraints and improving terrain judgment for features like ramps.
Summary updated