Explore-Bench: Data Sets, Metrics and Evaluations for Frontier-based and Deep-reinforcement-learning-based Autonomous Exploration

Yuanfan Xu, Jincheng Yu, Jiahao Tang, Jiantao Qiu, Jian Wang, Yuan Shen, Yu Wang, Huazhong Yang

Abstract

Explore-Bench is a comprehensive benchmark designed to standardize the evaluation of autonomous robotic exploration. It addresses the lack of unified datasets and metrics by providing a three-level evaluation platform, including a custom grid-based simulator optimized for Deep Reinforcement Learning (DRL) that operates 12x faster than standard physics-based simulators. This platform enables seamless transitions from high-speed training to realistic physics and real-world robot testing.

Introduction

Autonomous exploration—the ability of robots to navigate and map unknown environments—is a cornerstone of robotics. However, the field currently suffers from a lack of standardized evaluation, with researchers often testing algorithms in custom, cherry-picked scenarios. Furthermore, the rise of Deep Reinforcement Learning (DRL) has introduced a demand for millions of training interactions, which are computationally prohibitive in high-fidelity simulators like Gazebo. Explore-Bench bridges this gap by offering a tiered simulation-to-reality pipeline and principled metrics for both single and multi-robot systems.

The 3-Level Evaluation Platform

The core of Explore-Bench is its unified data flow, which standardizes inputs (merged 2D maps and robot poses) and outputs (goal coordinates). This is implemented across three distinct levels:

  • Level-0 (Fast Grid-Based Simulator): Specifically designed for DRL training, this level bypasses computationally expensive sensor rendering and SLAM by directly extracting local ground-truth maps. It is 12x faster than Gazebo and 36x faster than real-world testing.
  • Level-1 (Gazebo): A realistic physics simulator for high-fidelity testing of sensor rendering and motion dynamics.
  • Level-2 (Real Environment): A remotely accessible physical testbed using real robots (Xtark) and LiDAR for final validation.

Standardized Datasets and Metrics

Explore-Bench provides a rich set of environments built from architectural primitives such as loops, narrow corridors, and multi-room layouts.

Diverse simulation environments (a-f) used in Explore-Bench, ranging from simple loops to complex multi-room layouts.

The benchmark introduces specific metrics to evaluate performance beyond simple completion time:

  • Efficiency: Measured by $T_{total}$ (time to 99% map coverage) and $T_{topo}$ (time to 90% coverage, representing basic topological understanding).
  • Collaboration: Evaluated via Load Balance (standard deviation of area explored per robot) and Overlap Ratio (percentage of redundant exploration).

Key Insights and Results

Evaluation across the levels showed consistent algorithm rankings, validating the Level-0 simulator as a reliable proxy for real-world performance.

  • Algorithm Performance: Potential Field methods performed best overall, naturally balancing travel distance with information gain and facilitating multi-robot spreading through repulsive forces.
  • DRL Challenges: While DRL showed promise in collaboration (low overlap), it struggled with efficiency. The lack of an integrated local planner often caused robots to misinterpret high-level commands, leading to inefficient "jittering" motions.
  • Topological Understanding: The distinction between 90% and 99% coverage revealed that many algorithms spend excessive time backtracking to scan tiny corners after the main structure is already understood—a critical insight for time-sensitive missions like search and rescue.

Conclusion

Explore-Bench provides the necessary infrastructure for rigorous, scalable testing of exploration algorithms. By reducing the computational barrier for DRL and providing standardized metrics for multi-robot cooperation, it offers a path toward more robust autonomous systems. Future work will extend the benchmark to include 3D subterranean and outdoor environments.

Summary updated

BibTeX
@inproceedings{xu2022explore,
  author = {Yuanfan Xu and Jincheng Yu and Jiahao Tang and Jiantao Qiu and Jian Wang and Yuan Shen and Yu Wang and Huazhong Yang},
  title = {{Explore-Bench: Data Sets, Metrics and Evaluations for Frontier-based and Deep-reinforcement-learning-based Autonomous Exploration}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2022},
  doi = {10.1109/icra46639.2022.9812344},
}
Figure