Reconstruction of High-Precision Semantic Map

Xinyuan Tu, Jian Zhang, Runhao Luo, Kai Wang, Qingji Zeng, Yu Zhou, Yao Yu, Sidan DuSensors 2020 · DOI

TL;DR

This paper introduces a novel real-time system for creating high-precision 3D semantic maps from LiDAR data. It combines an efficient line-of-sight algorithm for surface reconstruction with an attention-based network for fusing semantic information, achieving both speed and accuracy for robotic navigation.

What Problem Does This Paper Address?

Creating detailed and semantically rich 3D maps in real-time is crucial for autonomous robots to navigate unfamiliar environments. While LiDAR sensors provide highly accurate depth information, processing this massive amount of data for both precise surface reconstruction and accurate semantic segmentation simultaneously is a significant challenge. Existing methods often struggle with either real-time performance, accuracy, or the integration of both surface and semantic information, especially when dealing with noisy LiDAR data. This paper aims to bridge this gap by developing a system that can efficiently reconstruct smooth surfaces and fuse semantic labels in real-time.

Methods

The proposed system integrates two main components: 3D surface reconstruction and semantic fusion.

  • 3D Surface Reconstruction: The core of the reconstruction is a novel voxel-based line-of-sight update algorithm. Instead of relying on depth images, this method directly processes unorganized LiDAR point clouds. It traces lines of sight from the sensor to the points, identifying and updating voxels along these paths. This approach incrementally updates a Truncated Signed Distance Field (TSDF) and is designed to reduce noise and handle the characteristics of LiDAR data. A dynamic weighting mechanism is also employed to account for varying noise levels in points at different distances.
  • Semantic Fusion: To achieve accurate semantic segmentation, the paper introduces an online attention-based spatial and temporal feature fusion network. This network takes inputs from:
  • Image Features: Derived from a 2D semantic segmentation network applied to RGB images.
  • Voxel Normals: Calculated from the reconstructed SDF.
  • Sensor Position: To evaluate observation effectiveness. This information is processed by an Observation Adaptive Network (OAN), which uses GRUs to learn observation effectiveness. Subsequently, an Attention Based Spatial Fusion Network (ABSFN) explicitly measures correlations between a voxel and its neighbors using self-attention, allowing for more robust semantic labeling by considering spatial context and temporal consistency. Parallel computation is leveraged throughout the system to ensure real-time performance.

Datasets

The system was evaluated on three datasets:

  • CARLA Dataset: A virtual dataset generated using the CARLA simulator, providing RGB images and LiDAR point clouds with ground truth for both semantic segmentation and 3D reconstruction. This dataset was used for detailed quantitative analysis.
  • Apollo Dataset: A real-world dataset known for its complex environments and noise. This dataset was used to evaluate the system's performance on challenging real-world scenarios.
  • VLP-16 Dataset: A custom dataset collected by the authors using a VLP-16 LiDAR sensor and two cameras, demonstrating the system's capabilities on collected real-world data.

Key Insights and Ideas

  • Line-of-Sight for LiDAR: Directly using a line-of-sight approach for TSDF updates, rather than relying on projected depth images, is a more natural and effective way to process unorganized LiDAR point clouds for surface reconstruction.
  • Adaptive Observation Evaluation: Quantifying the effectiveness of observations based on sensor position and voxel normals (via OAN) helps to filter out redundant or unreliable data, improving the robustness of semantic fusion.
  • Attention for Spatial Context: Employing self-attention (ABSFN) to explicitly model relationships between voxels and their neighbors is crucial for overcoming the limitations of purely local feature fusion and achieving more accurate semantic segmentation, especially for thin objects and complex boundaries.
  • Real-time Integration: The successful integration of both high-precision surface reconstruction and accurate semantic segmentation into a single, real-time pipeline is a significant achievement.

Key Results

  • Accuracy: The method achieves high accuracy in both 3D reconstruction and semantic segmentation. On the CARLA dataset, it shows improvements in IoU (up to 22.01% mIoU over ICNET [15]) and better handling of blurred boundaries and thin objects. In reconstruction, it achieves 94.87% accuracy at a 0.2m depth threshold and 86.91% at 0.1m, outperforming OctoMap.
  • Speed: The system achieves real-time performance, processing data at approximately 10 Hz. The total processing time per frame is around 65 ms on average, which is significantly faster (up to four times) than comparable methods like [32] that also perform both reconstruction and segmentation.
  • Robustness: The system demonstrates good performance on both simulated and real-world datasets, including the challenging Apollo dataset, and is able to recover small objects and reduce noise effectively.

Limitations and Future Work

  • Pose Estimation Error: The paper acknowledges that reconstruction errors can be magnified by pose estimation errors, particularly at the edges of small objects. The overall error is reported as 0.019m on average in CARLA.
  • Future Work: While not explicitly detailed, the paper's success in integrating spatial and temporal information suggests potential for further improvements in handling dynamic scenes and more complex environments. The focus on real-time performance also opens doors for deployment in a wider range of robotic applications.

Summary updated

Abstract

We present a real-time Truncated Signed Distance Field (TSDF)-based three-dimensional (3D) semantic reconstruction for LiDAR point cloud, which achieves incremental surface reconstruction and highly accurate semantic segmentation. The high-precise 3D semantic reconstruction in real time on LiDAR data is important but challenging. Lighting Detection and Ranging (LiDAR) data with high accuracy is massive for 3D reconstruction. We so propose a line-of-sight algorithm to update implicit surface incrementally. Meanwhile, in order to use more semantic information effectively, an online attention-based spatial and temporal feature fusion method is proposed, which is well integrated into the reconstruction system. We implement parallel computation in the reconstruction and semantic fusion process, which achieves real-time performance. We demonstrate our approach on the CARLA dataset, Apollo dataset, and our dataset. When compared with the state-of-art mapping methods, our method has a great advantage in terms of both quality and speed, which meets the needs of robotic mapping and navigation.
BibTeX
@inproceedings{tu2020reconstruction,
  author = {Xinyuan Tu and Jian Zhang and Runhao Luo and Kai Wang and Qingji Zeng and Yu Zhou and Yao Yu and Sidan Du},
  title = {{Reconstruction of High-Precision Semantic Map}},
  booktitle = {Sensors},
  year = {2020},
  doi = {10.3390/s20216264},
}
Figure