Semantic 3D Occupancy Mapping through Efficient High Order CRFs

Shichao Yang, Yulan Huang, Sebastian Scherer

Abstract

Semantic 3D mapping is essential for applications ranging from robot navigation to virtual interaction. While significant progress has been made in semantic segmentation and geometric 3D mapping independently, combining them for accurate, large-scale mapping remains a challenge. This paper proposes an incremental, near real-time semantic mapping system. We utilize a 3D scrolling occupancy grid to represent the environment efficiently, keeping memory and computation bounded. Our approach leverages CNN-based segmentation as a prior and optimizes 3D grid labels through a novel hierarchical Conditional Random Field (CRF) model. By using superpixels to enforce smoothness via robust $P^N$ high-order potentials and an efficient mean-field inference method, we achieve a 10% improvement in segmentation accuracy on the KITTI dataset compared to existing systems.

The Challenge of Large-Scale Semantic Mapping

Robots operating in the real world require both geometric awareness (where obstacles are) and semantic understanding (what those obstacles are). While 2D semantic segmentation has advanced through deep learning, it remains vulnerable to environmental factors like shadows and occlusions.

Existing 3D semantic mapping solutions often struggle to balance accuracy, scale, and real-time performance. Many are restricted to small indoor environments or require offline processing. This research aims to bridge this gap by providing an accurate system capable of handling large outdoor environments in near real-time.

System Architecture and Methodology

The proposed system integrates two primary pipelines: geometric mapping and semantic optimization.

The system pipeline: (a) Input images, (b) 3D occupancy map, (c) 2D CNN segmentation, (d) Superpixel generation, (e) CRF graph construction, and (f) Final semantic 3D map.

Geometric Mapping

To maintain a constant memory footprint, the system uses a scrolling occupancy grid. This grid maintains a fixed-size volume around the robot; as the robot moves, older data is cleared or archived. Dense stereo matching (ELAS) and camera tracking (ORB-SLAM) provide the underlying 3D structure and pose.

Semantic Labeling & Optimization

Initial semantic labels are generated via a Dilated CNN. These 2D priors are then projected into the 3D grid and optimized using a hierarchical CRF. The CRF model incorporates:

  • Unary Potentials: Derived from the CNN's initial predictions.
  • Pairwise Potentials: Encourage label consistency between adjacent voxels with similar visual and spatial features.
  • High-Order Potentials ($P^N$ Potts Model): By grouping voxels into superpixels, the model enforces regional consistency. This high-order approach is more robust than voxel-wise analysis alone, as it allows the system to "correct" 2D errors using 3D spatial context.

Key Insights: 3D Context Corrects 2D Ambiguity

A critical finding of this work is that 3D geometric consistency can resolve errors made by 2D segmentors. For instance, a fence obscured by shadows might be misclassified as vegetation in a single 2D frame. However, the CRF optimization recognizes the fence as a continuous, thin vertical structure in 3D space, allowing it to override the incorrect 2D label.

Results and Performance

The system was benchmarked on the KITTI dataset, focusing on 11 object classes. The hierarchical CRF approach yielded a 10% boost in global pixel accuracy and a 15% increase in Mean Intersection over Union (mIoU) compared to previous state-of-the-art 3D mapping methods.

Qualitative results showing the system's ability to accurately segment complex urban environments and small structures like poles (circled in red).

Significant improvements were noted in difficult categories: accuracy for "Fence" increased by 36.3% and "Pole" by 10.3%. Computationally, the grid optimization runs at approximately 2Hz on a standard CPU, which is sufficient for incremental updates during autonomous navigation.

Conclusion and Future Work

By combining a memory-efficient scrolling grid with high-order CRF inference, this system provides a robust framework for real-time semantic mapping. Future work will focus on increasing grid resolution for finer detail and porting the inference engine to GPUs to achieve frame rates exceeding 10Hz.

Summary updated

Abstract

Semantic 3D mapping can be used for many applications such as robot navigation and virtual interaction. In recent years, there has been great progress in semantic segmentation and geometric 3D mapping. However, it is still challenging to combine these two tasks for accurate and large-scale semantic mapping from images. In the paper, we propose an incremental and (near) real-time semantic mapping system. A 3D scrolling occupancy grid map is built to represent the world, which is memory and computationally efficient and bounded for large scale environments. We utilize the CNN segmentation as prior prediction and further optimize 3D grid labels through a novel CRF model. Superpixels are utilized to enforce smoothness and form robust P N high order potential. An efficient mean field inference is developed for the graph optimization. We evaluate our system on the KITTI dataset and improve the segmentation accuracy by 10% over existing systems.
BibTeX
@misc{yang2017semantic,
  author = {Shichao Yang and Yulan Huang and Sebastian Scherer},
  title = {{Semantic 3D Occupancy Mapping through Efficient High Order CRFs}},
  year = {2017},
  eprint = {1707.07388},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1707.07388},
}
Figure