Real-Time Metric-Semantic Mapping for Autonomous Navigation in Outdoor Environments

Jianhao Jiao, Ruoyu Geng, Yuanhang Li, Ren Xin, Bowen Yang, Jin Wu, Lujia Wang, Ming Liu, Rui Fan, Dimitrios Kanoulas

Abstract

Autonomous navigation in complex outdoor environments requires more than just geometric obstacle detection; it necessitates a semantic understanding of the surroundings to distinguish between traversable roads and restricted areas like sidewalks or vegetation. This paper presents a high-performance metric-semantic mapping system that fuses LiDAR, camera, and inertial data. By leveraging GPU acceleration and a Truncated Signed Distance Function (TSDF) representation, the system achieves real-time performance, processing frames in under 7ms. A recursive Bayesian update ensures temporal consistency of semantic labels, while a non-projective distance calculation improves reconstruction accuracy. Experimental results on public benchmarks and custom campus datasets demonstrate the system's efficiency and reliability for large-scale autonomous navigation.

Introduction

Most autonomous robots navigate using "geometric maps"—3D clouds of points or voxels that identify physical obstacles. However, geometry alone is insufficient for complex outdoor navigation. To a LiDAR sensor, a flat paved road, a flat sidewalk, and a patch of flat grass look nearly identical. To follow traffic rules and navigate safely, a robot needs semantic information: it must distinguish between a road meant for vehicles and a sidewalk meant for pedestrians.

Existing metric-semantic mapping systems often struggle with computational speed, scalability to large outdoor environments, and the temporal consistency of 2D labels when projected into 3D space. This research addresses these challenges through a modular, GPU-accelerated framework.

System Architecture and Methods

The proposed system, COBRA, utilizes a LiDAR-Visual-Inertial (LVI) setup to create dense, labeled 3D maps. The architecture is designed for maximum throughput and accuracy.

Figure 3: Overview of the COBRA mapping pipeline, showing the fusion of Camera, LiDAR, and IMU data into a metric-semantic mesh.

As illustrated in the system overview, the pipeline consists of several key components:

  • State Estimation: An EKF-based LVI odometry tracks the robot's position with high precision.
  • Semantic Segmentation: A CNN processes 2D camera images to label pixels (e.g., "road," "vegetation," "building").
  • TSDF-Based Mapping: Instead of simple point clouds, the system uses a Truncated Signed Distance Function (TSDF) to represent surfaces, allowing for sub-voxel resolution and smoother meshes.
  • GPU Acceleration: The entire mapping pipeline—from point cloud processing to mesh updates—is implemented in parallel on the GPU, keeping processing time under 7ms per frame.
  • Recursive Bayesian Update: To handle noisy 2D labels, a Bayesian filter tracks the probability of labels for every 3D voxel over time, ensuring that transient errors do not corrupt the global map.

Experimental Results

The system was evaluated on established benchmarks (SemanticKITTI, SemanticUSL) and a custom campus dataset comprising 15 sequences with diverse terrains.

Figure 1: Large-scale metric-semantic map generated by the system, demonstrating clear differentiation between drivable roads and restricted zones.

Performance and Accuracy

On a desktop (RTX 3080Ti), the system processes frames in 2.4ms to 6.5ms. On embedded platforms like the Nvidia Jetson Orin, it maintains a viable 20-30ms latency. In tests on SemanticKITTI, the system achieved lower Reconstruction Error (RE) and Chamfer Distance (CD) compared to CPU-based baselines like VoxBlox.

Real-World Validation

Deployment on an autonomous vehicle moving at 3m/s confirmed the system's utility. While a robot using only geometric data might attempt to drive across a flat lawn, the semantic layer correctly identifies grass as a non-drivable zone, ensuring safe and rule-compliant navigation.

Discussion and Future Work

A key insight from this work is that semantic labels act as a critical safety layer for traversability analysis. By using a "non-projective" distance calculation that considers local planarity, the system achieves higher reconstruction accuracy than standard projective methods.

However, limitations remain. The system's heavy reliance on GPU memory may require "submap" strategies for city-scale deployment. Additionally, the lack of a global loop closure mechanism means that small odometry drifts can accumulate over very long distances. Future work will focus on integrating loop closure and exploring kernel-based methods to further improve spatio-temporal semantic consistency.

Summary updated

Abstract

The creation of a metric-semantic map, which encodes human-prior knowledge, represents a high-level abstraction of environments. However, constructing such a map poses challenges related to the fusion of multi-modal sensor data, the attainment of real-time mapping performance, and the preservation of structural and semantic information consistency. In this paper, we introduce an online metric-semantic mapping system that utilizes LiDAR-Visual-Inertial sensing to generate a global metric-semantic mesh map of large-scale outdoor environments. Leveraging GPU acceleration, our mapping process achieves exceptional speed, with frame processing taking less than 7ms, regardless of scenario scale. Furthermore, we seamlessly integrate the resultant map into a real-world navigation system, enabling metric-semantic-based terrain assessment and autonomous point-to-point navigation within a campus environment. Through extensive experiments conducted on both publicly available and self-collected datasets comprising 24 sequences, we demonstrate the effectiveness of our mapping and navigation methodologies. Code has been publicly released: https://github.com/gogojjh/cobra
BibTeX
@inproceedings{jiao2024real,
  author = {Jianhao Jiao and Ruoyu Geng and Yuanhang Li and Ren Xin and Bowen Yang and Jin Wu and Lujia Wang and Ming Liu and Rui Fan and Dimitrios Kanoulas},
  title = {{Real-Time Metric-Semantic Mapping for Autonomous Navigation in Outdoor Environments}},
  booktitle = {IEEE Trans Autom. Sci. Eng},
  year = {2024},
}
Figure