Abstract
Autonomous exploration of large, unknown environments remains a significant challenge for mobile robotics due to the curse of dimensionality and sparse rewards. This paper proposes a hierarchical reinforcement learning (RL) framework that decomposes the exploration task into three manageable sub-tasks: Target Selection, Target Search, and Region Exploration. By utilizing specialized algorithms for long-range navigation and local granular searching, the proposed model achieves significantly higher efficiency in discovering Areas of Interest (AoIs) compared to traditional curiosity-driven and hardcoded search strategies.
Introduction
Autonomous exploration is essential for critical applications such as search and rescue, environmental monitoring, and mapping. While Simultaneous Localization and Mapping (SLAM) is well-established, applying RL to these tasks often fails due to the "state-space explosion." Navigating large, unknown areas provides sparse feedback, causing agents to become lost before learning useful policies. Furthermore, most RL solutions assume a pre-defined destination, which is inapplicable when the goal is to discover unknown features like survivors or illegal activity sites.
Methodology
The proposed framework employs a "divide and conquer" strategy, breaking the exploration cycle into three distinct phases:
- Target Selection (Optimization): The map is divided into nine regions. A target region is selected using an optimization equation based on distance, current coverage, and the density of AoIs already discovered.
- Target Search (DDQN + LSTM): The agent navigates toward the selected region using a Double Deep Q-Network (DDQN) with an LSTM layer. To manage state space, the agent's view is restricted to a 25x25 local map. It is rewarded for hitting AoIs encountered during transit, encouraging opportunistic discovery.
- Region Exploration (A2C): Upon arrival, the agent switches to an Advantage Actor-Critic (A2C) algorithm to perform granular searching within the local region, receiving rewards for visiting unknown cells and covering AoIs efficiently.
Experimental Results
The model was evaluated in a simulated grid-based environment against three baselines: curiosity-driven exploration, a hardcoded zigzag pattern, and a random-action framework. Efficiency was measured by the number of steps required to reach specific AoI coverage thresholds.
As shown in the results, the proposed model reached 70% coverage approximately 14% faster than the hardcoded zigzag policy. Notably, the curiosity-driven baseline failed to reach coverage thresholds within the 40,000-step limit, highlighting that structured hierarchy is more effective for large-scale mapping than pure intrinsic motivation.
Key Insights
- Hierarchy Beats Curiosity: Structured sub-tasks prevent the agent from becoming overwhelmed by the global state space.
- Local State Management: Restricting the agent's vision to local maps during transit effectively bypasses the dimensionality curse.
- Opportunistic Navigation: Rewarding discovery during the search phase transforms simple point-to-point navigation into active observation.
Conclusion and Future Work
The hierarchical approach demonstrates superior efficiency in unknown environment exploration. Future work will focus on dynamic map partitioning for even larger environments and extending the framework to multi-agent systems for parallel exploration. Real-world applications, including illegal mining detection in the Amazon rainforest and search and rescue operations, are currently being planned.
Summary updated