Where to Explore Next? ExHistCNN for History-aware Autonomous 3D Exploration

Yiming Wang, Alessio Del Bue

Abstract

In this work we address the problem of autonomous 3D exploration of an unknown indoor environment using a depth camera. We cast the problem as the estimation of the Next Best View (NBV) that maximises the coverage of the unknown area. We do this by re-formulating NBV estimation as a classification problem and we propose a novel learning-based metric that encodes both, the current 3D observation (a depth frame) and the history of the ongoing reconstruction. One of the major contributions of this work is about introducing a new representation for the 3D reconstruction history as an auxiliary utility map which is efficiently coupled with the current depth observation. With both pieces of information, we train a light-weight CNN, named ExHistCNN, that estimates the NBV as a set of directions towards which the depth sensor finds most unexplored areas. We perform extensive evaluation on both synthetic and real room scans demonstrating that the proposed ExHistCNN is able to approach the exploration performance of an oracle using the complete knowledge of the 3D environment.

Introduction

Autonomous 3D reconstruction of indoor environments is a critical task for robotics, yet it remains challenging due to the "Next Best View" (NBV) problem. A robot must decide where to move next to maximize the coverage of unexplored areas. Traditional approaches often suffer from two main issues: computational inefficiency when processing full 3D volumetric maps and "history blindness," where a robot lacks a memory of previously visited locations, leading to redundant exploration.

Methodology

The proposed system, ExHistCNN, addresses these challenges by reframing NBV estimation as a classification problem. Instead of calculating continuous coordinates, the network selects the optimal movement from four discrete directions: up, down, left, or right.

Figure 1: Overview of the autonomous 3D exploration pipeline, integrating data capturing, 3D memory representation, and the ExHistCNN for NBV estimation.

The Utility Map

A core innovation of this work is the Utility Map. The robot maintains a 3D "OctoMap" (a voxel-based representation) categorized into Free, Occupied, or Unknown voxels. Using ray-tracing, the system projects these "Unknown" areas into a 2D binary image. In this map, a pixel value of 1 indicates an unexplored area, while 0 represents previously seen space. This allows the robot to incorporate reconstruction history without the overhead of complex 3D data.

ExHistCNN Architecture

ExHistCNN is a light-weight 2D CNN designed to process two primary inputs: the current depth frame and the utility map. The researchers explored various input formats, including stacking images and partitioning the utility map into directional zones (e.g., CNN5D) to explicitly guide the network toward unexplored regions.

Figure 2: The ExHistCNN architecture and the various input data formats evaluated, including depth maps and partitioned utility maps.

Evaluation and Results

The model was trained using a specialized pipeline involving synthetic indoor environments (SUNCG) and real-world scans (Matterport3D). A dataset of 100,000 samples per direction class was generated using an "Oracle" strategy—a script with full knowledge of the environment—to provide ground-truth labels.

Key findings include:

  • Near-Oracle Performance: ExHistCNN achieved exploration efficiency comparable to the Oracle strategy in synthetic environments.
  • Computational Efficiency: The system is highly responsive, requiring only ~0.028 seconds per step to determine the NBV.
  • Early Coverage Gains: The model shows a consistent boost in scene coverage during the initial stages of exploration compared to traditional hand-crafted metrics.

Conclusion

ExHistCNN demonstrates that 2D representations of 3D frontiers are sufficient for high-quality autonomous exploration. By coupling current observations with a history-aware utility map, the system achieves efficient and effective mapping. Future work will focus on improving domain adaptation for real-world sensor noise and developing learned stopping conditions to signal the completion of a reconstruction task.

Summary updated

Abstract

In this work we address the problem of autonomous 3D exploration of an unknown indoor environment using a depth camera. We cast the problem as the estimation of the Next Best View (NBV) that maximises the coverage of the unknown area. We do this by re-formulating NBV estimation as a classification problem and we propose a novel learning-based metric that encodes both, the current 3D observation (a depth frame) and the history of the ongoing reconstruction. One of the major contributions of this work is about introducing a new representation for the 3D reconstruction history as an auxiliary utility map which is efficiently coupled with the current depth observation. With both pieces of information, we train a light-weight CNN, named ExHistCNN, that estimates the NBV as a set of directions towards which the depth sensor finds most unexplored areas. We perform extensive evaluation on both synthetic and real room scans demonstrating that the proposed ExHistCNN is able to approach the exploration performance of an oracle using the complete knowledge of the 3D environment.
BibTeX
@inproceedings{wang2020where,
  author = {Yiming Wang and Alessio Del Bue},
  title = {{Where to Explore Next? ExHistCNN for History-aware Autonomous 3D Exploration}},
  booktitle = {European Conference on Computer Vision},
  year = {2020},
  url = {https://openreview.net/forum?id=22zPyJ6ilf},
}
Figure