Efficient Hierarchical Reinforcement Learning for Mapless Navigation With Predictive Neighbouring Space Scoring

Yan Gao, Jing Wu, Xintong Yang, Ze Ji

Abstract

Solving reinforcement learning (RL)-based mapless navigation tasks is challenging due to their sparse reward and long decision horizon nature. Hierarchical reinforcement learning (HRL) has the ability to leverage knowledge at different abstract levels and is thus preferred in complex mapless navigation tasks. However, it is computationally expensive and inefficient to learn navigation end-to-end from raw high-dimensional sensor data, such as Lidar or RGB cameras. The use of subgoals based on a compact intermediate representation is therefore preferred for dimension reduction. This work proposes an efficient HRL-based framework to achieve this with a novel scoring method, named Predictive Neighbouring Space Scoring (PNSS). The PNSS model estimates the explorable space for a given position of interest based on the current robot observation. The PNSS values for a few candidate positions around the robot provide a compact and informative state representation for subgoal selection. We study the effects of different candidate position layouts and demonstrate that our layout design facilitates higher performances in longer-range tasks. Moreover, a penalty term is introduced in the reward function for the high-level (HL) policy, so that the subgoal selection process takes the performance of the low-level (LL) policy into consideration. Comprehensive evaluations demonstrate that using the proposed PNSS module consistently improves performances over the use of Lidar only or Lidar and encoded RGB features.

The Challenge of Mapless Navigation

Mapless navigation represents a critical capability for autonomous robots operating in unstructured environments such as homes, warehouses, or disaster zones. Unlike traditional methods that rely on pre-built maps, mapless agents must navigate using real-time sensor data (e.g., Lidar and RGB cameras). However, standard Deep Reinforcement Learning (DRL) approaches often struggle with two primary issues:

  1. Sparse Rewards: In long-range tasks, a robot may wander extensively before receiving feedback, leading to inefficient learning.
  2. Local Minima: Robots frequently become trapped by obstacles (like U-shaped walls) because their immediate policy prioritizes moving directly toward the goal, even when the path is blocked.

Hierarchical Reinforcement Learning (HRL) addresses these by splitting the task: a High-Level (HL) policy selects "subgoals," while a Low-Level (LL) policy handles motor control to reach them. However, HRL still faces efficiency hurdles when processing high-dimensional raw sensor data.

The PNSS Framework

To overcome these limitations, this research introduces the Predictive Neighbouring Space Scoring (PNSS) module. Instead of feeding raw RGB images directly into the RL agent, the framework uses a UNet-based architecture to predict the "explorable space"—the proportion of free versus occupied cells—around potential subgoals. This provides a compact, informative score that guides the High-Level policy.

Overview of the hierarchical reinforcement learning architecture featuring the PNSS model and dual-level policy structure.

The hierarchical structure operates as follows:

  • High-Level (HL) Policy: Utilizes a Deep Q-Network (DQN) to select subgoals from a forward-facing 3x3 grid. A key innovation is the inclusion of a reachability penalty; if the LL policy fails to reach a subgoal due to collision or timeout, the HL policy is penalized, forcing the "brain" to avoid impossible targets.
  • Low-Level (LL) Policy: Employs Deep Deterministic Policy Gradient (DDPG) to control wheel velocities and execute the movement to the selected subgoal.

Predictive Neighbouring Space Scoring (PNSS)

The core "aha moment" of this work is that a subgoal's value is best represented by its potential for exploration rather than just its coordinates. By scoring the "openness" of a point, the HL policy learns to prioritize paths that lead to open spaces rather than cluttered corners.

Visualization of how PNSS values represent explorable space (openness) around candidate subgoals.

The authors also optimized the subgoal layout. While robots often have 360-degree Lidar, humans navigate by looking ahead. The study found that a forward-biased 3x3 grid encouraged the robot to look further ahead, providing richer semantic information for long-range navigation.

Experimental Results

Evaluations were conducted in the iGibson simulator, using 23 distinct photo-realistic environments (houses, offices, cafes). The PNSS-based HRL framework was compared against non-hierarchical baselines and state-of-the-art HRL methods like HiRO.

  • Success Rates: In difficult long-range tasks (8–10 meters), the proposed method achieved success rates nearly 40% higher than discrete non-hierarchical methods.
  • Obstacle Avoidance: Qualitative results showed the robot successfully navigating around complex "U" traps that caused baseline agents to get stuck indefinitely.
  • Efficiency: Using PNSS scores as an intermediate representation proved more efficient than using raw Lidar or encoded RGB features alone, leading to more stable training and better generalization to unseen environments.

Conclusion and Future Work

While the PNSS module significantly improves navigation, challenges remain. Lidar can still miss thin obstacles like chair legs, and the current model is optimized for static environments. Future work will focus on narrowing the simulation-to-reality gap by testing on physical robots and accounting for dynamic obstacles like moving humans. Additionally, the computational overhead of predicting PNSS values (approx. 0.358s per decision) presents an opportunity for further optimization in high-speed robotics applications.

Summary updated

Abstract

Solving reinforcement learning (RL)-based mapless navigation tasks is challenging due to their sparse reward and long decision horizon nature. Hierarchical reinforcement learning (HRL) has the ability to leverage knowledge at different abstract levels and is thus preferred in complex mapless navigation tasks. However, it is computationally expensive and inefficient to learn navigation end-to-end from raw high-dimensional sensor data, such as Lidar or RGB cameras. The use of subgoals based on a compact intermediate representation is therefore preferred for dimension reduction. This work proposes an efficient HRL-based framework to achieve this with a novel scoring method, named Predictive Neighbouring Space Scoring (PNSS). The PNSS model estimates the explorable space for a given position of interest based on the current robot observation. The PNSS values for a few candidate positions around the robot provide a compact and informative state representation for subgoal selection. We study the effects of different candidate position layouts and demonstrate that our layout design facilitates higher performances in longer-range tasks. Moreover, a penalty term is introduced in the reward function for the high-level (HL) policy, so that the subgoal selection process takes the performance of the low-level (LL) policy into consideration. Comprehensive evaluations demonstrate that using the proposed PNSS module consistently improves performances over the use of Lidar only or Lidar and encoded RGB features <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —This paper seeks to improve robot mapless navigation capabilities where the robot is expected to navigate to a goal location without knowing the map of the environment. This ability is highly demanded in many applications that require autonomous operations in unstructured environments, including both indoor and outdoor scenarios, involving tasks such as service robots for domestic and public environments, logistics in industrial warehouses, urban search and rescue missions, and disaster relief efforts, where detailed and accurate maps are difficult to obtain in advance. In this work, we focus on reinforcement learning-based mapless navigation. It is known that such methods struggle in complex long-range tasks, e.g. stuck in a local region by multiple objects. Therefore, this paper proposes a novel mapless navigation method inspired by human navigation behaviours. We enable a robot to split a long-range navigation task into multiple segments, by selecting and navigating to short-term goals. These subgoals are selected each time from a number of candidate positions located around the robot. The process stops when the robot reaches the final target location. When selecting a short-term goal, we use a deep neural network to predict the openness around each candidate subgoal position, named the Predictive Neighbouring Space Scoring (PNSS), from raw images and Lidar scans. In addition, we study the effects of different arrangements of candidate subgoal locations and select the optimal one. Experiments conducted in photo-realistic simulation environments demonstrate the effectiveness of our method, showcasing superior performance over baselines. It is worth noting that our agent is only trained in domestic environments using the iGibson simulator. For applications in other environments, additional training in more representative settings specific to corresponding scenarios will be necessary. In the future, our intention is to validate our methods in complex real-world environments and narrow the simulation-to-reality gap for long-horizon navigation tasks.
BibTeX
@article{gao2023efficient,
  author = {Yan Gao and Jing Wu and Xintong Yang and Ze Ji},
  title = {{Efficient Hierarchical Reinforcement Learning for Mapless Navigation With Predictive Neighbouring Space Scoring}},
  journal = {IEEE Transactions on Automation Science and Engineering},
  year = {2023},
  doi = {10.1109/tase.2023.3312237},
}
Figure