Abstract
Solving reinforcement learning (RL)-based mapless navigation tasks is challenging due to their sparse reward and long decision horizon nature. Hierarchical reinforcement learning (HRL) has the ability to leverage knowledge at different abstract levels and is thus preferred in complex mapless navigation tasks. However, it is computationally expensive and inefficient to learn navigation end-to-end from raw high-dimensional sensor data, such as Lidar or RGB cameras. The use of subgoals based on a compact intermediate representation is therefore preferred for dimension reduction. This work proposes an efficient HRL-based framework to achieve this with a novel scoring method, named Predictive Neighbouring Space Scoring (PNSS). The PNSS model estimates the explorable space for a given position of interest based on the current robot observation. The PNSS values for a few candidate positions around the robot provide a compact and informative state representation for subgoal selection. We study the effects of different candidate position layouts and demonstrate that our layout design facilitates higher performances in longer-range tasks. Moreover, a penalty term is introduced in the reward function for the high-level (HL) policy, so that the subgoal selection process takes the performance of the low-level (LL) policy into consideration. Comprehensive evaluations demonstrate that using the proposed PNSS module consistently improves performances over the use of Lidar only or Lidar and encoded RGB features.
The Challenge of Mapless Navigation
Mapless navigation represents a critical capability for autonomous robots operating in unstructured environments such as homes, warehouses, or disaster zones. Unlike traditional methods that rely on pre-built maps, mapless agents must navigate using real-time sensor data (e.g., Lidar and RGB cameras). However, standard Deep Reinforcement Learning (DRL) approaches often struggle with two primary issues:
- Sparse Rewards: In long-range tasks, a robot may wander extensively before receiving feedback, leading to inefficient learning.
- Local Minima: Robots frequently become trapped by obstacles (like U-shaped walls) because their immediate policy prioritizes moving directly toward the goal, even when the path is blocked.
Hierarchical Reinforcement Learning (HRL) addresses these by splitting the task: a High-Level (HL) policy selects "subgoals," while a Low-Level (LL) policy handles motor control to reach them. However, HRL still faces efficiency hurdles when processing high-dimensional raw sensor data.
The PNSS Framework
To overcome these limitations, this research introduces the Predictive Neighbouring Space Scoring (PNSS) module. Instead of feeding raw RGB images directly into the RL agent, the framework uses a UNet-based architecture to predict the "explorable space"—the proportion of free versus occupied cells—around potential subgoals. This provides a compact, informative score that guides the High-Level policy.
The hierarchical structure operates as follows:
- High-Level (HL) Policy: Utilizes a Deep Q-Network (DQN) to select subgoals from a forward-facing 3x3 grid. A key innovation is the inclusion of a reachability penalty; if the LL policy fails to reach a subgoal due to collision or timeout, the HL policy is penalized, forcing the "brain" to avoid impossible targets.
- Low-Level (LL) Policy: Employs Deep Deterministic Policy Gradient (DDPG) to control wheel velocities and execute the movement to the selected subgoal.
Predictive Neighbouring Space Scoring (PNSS)
The core "aha moment" of this work is that a subgoal's value is best represented by its potential for exploration rather than just its coordinates. By scoring the "openness" of a point, the HL policy learns to prioritize paths that lead to open spaces rather than cluttered corners.
The authors also optimized the subgoal layout. While robots often have 360-degree Lidar, humans navigate by looking ahead. The study found that a forward-biased 3x3 grid encouraged the robot to look further ahead, providing richer semantic information for long-range navigation.
Experimental Results
Evaluations were conducted in the iGibson simulator, using 23 distinct photo-realistic environments (houses, offices, cafes). The PNSS-based HRL framework was compared against non-hierarchical baselines and state-of-the-art HRL methods like HiRO.
- Success Rates: In difficult long-range tasks (8–10 meters), the proposed method achieved success rates nearly 40% higher than discrete non-hierarchical methods.
- Obstacle Avoidance: Qualitative results showed the robot successfully navigating around complex "U" traps that caused baseline agents to get stuck indefinitely.
- Efficiency: Using PNSS scores as an intermediate representation proved more efficient than using raw Lidar or encoded RGB features alone, leading to more stable training and better generalization to unseen environments.
Conclusion and Future Work
While the PNSS module significantly improves navigation, challenges remain. Lidar can still miss thin obstacles like chair legs, and the current model is optimized for static environments. Future work will focus on narrowing the simulation-to-reality gap by testing on physical robots and accounting for dynamic obstacles like moving humans. Additionally, the computational overhead of predicting PNSS values (approx. 0.358s per decision) presents an opportunity for further optimization in high-speed robotics applications.
Summary updated