Introduction
High-definition (HD) maps are a cornerstone of safe autonomous navigation, providing vehicles with a detailed environmental prior. Traditionally, these maps are generated using expensive LiDAR-equipped survey vehicles and extensive manual annotation. However, the high cost and labor-intensive nature of this approach make it difficult to scale and maintain in rapidly changing urban environments. This paper presents a scalable alternative: a vision-only pipeline capable of generating large-scale 3D semantic maps using standard stereo camera systems.
Methodology
The proposed system bridges the gap between 2D image-based perception and 3D spatial understanding through a modular four-stage pipeline.
1. Visual Odometry (VO) Front-end
The system utilizes a direct sparse SLAM approach based on Stereo DSO. Unlike indirect methods that rely on sparse feature points (like corners), direct methods leverage the full intensity information of pixels. This results in a richer 3D point cloud that captures more environmental detail, which is crucial for subsequent semantic mapping.
2. Semantic Segmentation
A Deep Neural Network (DNN) is employed to generate 2D semantic labels for every pixel in the camera frames. While these networks are highly capable, their output in video sequences often suffers from "flicker" or inconsistent labeling due to motion blur, varying lighting conditions, and perspective changes.
3. Temporally Consistent Labeling (TCL)
The core innovation of this work is the Temporally Consistent Labeling (TCL) scheme. To resolve 2D inconsistencies, the system projects 3D points into multiple neighboring frames where they were observed. Instead of a simple majority vote, the system uses a weighted voting scheme based on inverse depth.
The authors observed that 2D segmentation accuracy is highest at mid-range distances. Points too far away are often misclassified due to low resolution, while points too close suffer from motion blur. By weighting votes by inverse depth and applying distance thresholds, the pipeline effectively filters out classification noise, leading to a stable 3D semantic representation.
4. Global Optimization
To ensure the map remains accurate over city-scale distances, the back-end integrates GNSS data and performs loop closure. This global optimization step corrects for the drift inherent in visual odometry, maintaining spatial consistency across thousands of kilometers.
Experimental Results
The system was evaluated on two primary datasets:
- KITTI-360: A 74 km suburban dataset. The authors established a new benchmark by fusing 3D LiDAR and 2D image ground truth to rigorously evaluate vision-based pipelines.
- Berlin Fleet Dataset: To demonstrate scalability, the system processed data from a fleet of vehicles covering 8,000 km of roadway in Berlin.
Performance Gains
The implementation of the TCL scheme resulted in a significant performance boost, increasing the Mean Intersection over Union (mIoU) on the KITTI-360 dataset from a baseline of 58.5% to 65.1%. Notably, the system showed marked improvements in identifying challenging, small-scale objects such as poles, traffic lights, and sidewalks—elements that are frequently misidentified in noisy 2D frames.
Discussion and Future Work
The results demonstrate that vision-only systems can achieve the scale required for city-wide HD mapping, a task previously dominated by LiDAR. However, several challenges remain. The current output is a sparse point cloud rather than a dense, "watertight" surface. Additionally, the map is not yet fully vectorized (e.g., containing lane connectivity and mathematical curve representations).
Future research will focus on transitioning toward 3D volumetric mapping. By combining this semantic pipeline with dense reconstruction techniques, the authors aim to create even more detailed and actionable environmental models for autonomous systems.
Summary updated