Where to Explore Next? ExHistCNN for History-Aware Autonomous 3D Exploration

Yiming Wang, Alessio Del Bue

Abstract

In this work we address the problem of autonomous 3D exploration of an unknown indoor environment using a depth camera. We cast the problem as the estimation of the Next Best View (NBV) that maximises the coverage of the unknown area. We do this by re-formulating NBV estimation as a classification problem and we propose a novel learning-based metric that encodes both, the current 3D observation (a depth frame) and the history of the ongoing reconstruction. One of the major contributions of this work is about introducing a new representation for the 3D reconstruction history as an auxiliary utility map which is efficiently coupled with the current depth observation. With both pieces of information, we train a light-weight CNN, named ExHistCNN, that estimates the NBV as a set of directions towards which the depth sensor finds most unexplored areas. We perform extensive evaluation on both synthetic and real room scans demonstrating that the proposed ExHistCNN is able to approach the exploration performance of an oracle using the complete knowledge of the 3D environment.

Introduction

When a robot is deployed in an unknown environment, it must decide where to point its sensors to build a comprehensive 3D map. This is known as the Next Best View (NBV) problem. Traditional approaches rely on hand-crafted geometric heuristics, which are computationally efficient but often lack the foresight to avoid local minima. Conversely, modern 3D deep learning approaches are powerful but computationally heavy and slow for real-time robotics.

The authors of this paper identify a critical gap: a single depth image lacks the context of what has already been explored. To solve this, they propose ExHistCNN, a system that integrates the history of the reconstruction into a lightweight 2D learning framework.

Methodology

ExHistCNN treats NBV estimation as a classification problem. The system architecture follows a loop: capturing data, updating a 3D scene reconstruction, and maintaining a 3D memory representation.

The overall pipeline of the ExHistCNN system, from data capture to NBV selection.

The Utility Map

The core innovation is the utility map. Instead of processing a complex 3D voxel grid, the system uses ray-tracing to project the current knowledge of the environment into a 2D binary image. In this map, pixels are marked as unexplored ('1') or visited ('0'). This allows the robot to "remember" its history without the overhead of 3D convolutions.

Architecture and Input Formats

The researchers designed a lightweight 2D CNN that takes the current depth frame and the utility map as inputs. They experimented with several data formats to find the most effective way to represent spatial history. The most successful variant, CNN5D, uses a depth map stacked with four partitioned utility maps (Up, Down, Left, and Right), which explicitly helps the network associate unexplored regions with specific movement directions.

The ExHistCNN architecture and the various input data formats tested, ranging from simple depth to partitioned utility maps.

Experiments and Results

The model was trained using an "Oracle"—a system with full knowledge of the environment—to provide ground-truth labels for the directions that reveal the most new surface area. Testing was performed on the SUNCG synthetic dataset and the Matterport3D real-world scan dataset.

  • Efficiency: ExHistCNN variants (CNN2D and CNN5D) achieved performance levels close to the Oracle, significantly outperforming models that lacked history awareness (CNNDepth).
  • Speed: The system is highly efficient, with an inference time of approximately 0.027 seconds. This is faster than many traditional geometric methods and substantially faster than 3D learning-based alternatives.
  • Generalization: While there is a slight performance gap when moving from synthetic to real-world data, the model remained the most efficient at covering ground during the initial phases of exploration.

Conclusion

ExHistCNN demonstrates that 3D exploration can be solved efficiently by encoding reconstruction history into 2D utility maps. By combining current depth observations with a partitioned memory of visited areas, robots can make "smarter" decisions about where to explore next without the need for heavy 3D computation. Future work will focus on domain adaptation to bridge the reality gap and expanding the action space beyond discrete directional choices.

Summary updated

Abstract

In this work we address the problem of autonomous 3D exploration of an unknown indoor environment using a depth camera. We cast the problem as the estimation of the Next Best View (NBV) that maximises the coverage of the unknown area. We do this by re-formulating NBV estimation as a classification problem and we propose a novel learning-based metric that encodes both, the current 3D observation (a depth frame) and the history of the ongoing reconstruction. One of the major contributions of this work is about introducing a new representation for the 3D reconstruction history as an auxiliary utility map which is efficiently coupled with the current depth observation. With both pieces of information, we train a light-weight CNN, named ExHistCNN, that estimates the NBV as a set of directions towards which the depth sensor finds most unexplored areas. We perform extensive evaluation on both synthetic and real room scans demonstrating that the proposed ExHistCNN is able to approach the exploration performance of an oracle using the complete knowledge of the 3D environment.
BibTeX
@inproceedings{wang2020where,
  author = {Yiming Wang and Alessio Del Bue},
  title = {{Where to Explore Next? ExHistCNN for History-Aware Autonomous 3D Exploration}},
  booktitle = {European Conference on Computer Vision},
  year = {2020},
  url = {https://openreview.net/forum?id=22zPyJ6ilf},
}
Figure