Abstract
Navigating unknown environments without prior maps remains a significant challenge for autonomous robots, often leading to local minima in complex topologies. We present the Skill Q-Network (SQN), a reinforcement learning architecture that learns an adaptive ensemble of latent skills. SQN decomposes navigation into perception, decision-making, and specialized skill execution. By incorporating a reachability-aware reward function, the model effectively balances goal-directed behavior with environment exploration. Our results demonstrate that SQN achieves a 98% success rate in maze environments and exhibits strong zero-shot generalization to unseen, large-scale subterranean caves, outperforming traditional monolithic RL baselines.
Introduction
Mapless navigation requires a robot to reach a target using only local observations, such as 2D range measurements. Standard Reinforcement Learning (RL) approaches typically employ a single, monolithic policy network. However, in environments with complex obstacles and dead ends, these models often fail to transition between exploring the space and navigating toward the goal. While Hierarchical RL (HRL) offers a potential solution, it frequently relies on manually defined sub-tasks. SQN addresses these limitations by discovering latent skills and learning an adaptive selection mechanism end-to-end.
Methodology
The SQN architecture is composed of four primary components designed to facilitate modular learning and adaptive behavior.
- Perception Module: This module processes 2D range measurements and relative goal coordinates using Multi-Layer Perceptrons (MLP). A Gated Recurrent Unit (GRU) is integrated to provide the agent with temporal memory, essential for navigating partially observable environments.
- Decision Module: Acting as a high-level manager, this module outputs a "skill decision" ($\eta_t^\pi$)—an attention score that determines the relative importance of each latent skill for the current state.
- Latent Skill Policies: Instead of a single output head, SQN utilizes $N$ parallel skill heads. Each head learns a distinct navigational strategy (e.g., wall-following or open-space traversal) through experience.
- Skill Ensemble: The final action is determined by aggregating the Q-values from all latent skills, weighted by the attention scores from the Decision Module.
Reachability-Aware Reward
A key innovation in SQN is the "Reachability-Aware" reward function. Traditional rewards based solely on goal distance can lead to "optimism bias," where a robot attempts to move through walls to reach a nearby goal. SQN provides a goal-seeking reward only if the target is within the robot's current line-of-sight (reachable).
When the goal is blocked, the reward function prioritizes an exploration term, incentivizing the robot to map new areas and find a path around obstacles.
Experimental Results
The model was trained in a 17x17m maze and evaluated across several Out-of-Distribution (OOD) environments, including "Column," "Forest," and a large-scale "SubT Cave."
- Success Rate and Efficiency: In the training maze, SQN achieved a 98% success rate, significantly higher than DQN and R2D2 baselines (~72%). Furthermore, SQN reached the goal using 42% fewer time steps on average.
- Zero-Shot Generalization: SQN maintained an 82% success rate in the complex SubT Cave environment (51x51m), whereas the DQN baseline's performance dropped to 38%. This suggests that the latent skills learned are generalizable strategies rather than environment-specific paths.
- Robustness: The architecture demonstrated high tolerance to sensor noise, maintaining over 80% success even with 60% uniform noise added to range measurements.
Conclusion
The Skill Q-Network demonstrates that functional modularity and reachability-aware rewards can significantly improve mapless navigation. By learning latent skills end-to-end, SQN avoids the need for manual task decomposition and achieves superior robustness in unseen environments. Future work will focus on bridging the sim-to-real gap and exploring dynamic skill allocation to further enhance navigational flexibility.
Summary updated