Skill Q-Network: Learning Adaptive Skill Ensemble for Mapless Navigation in Unknown Environments

Hyunki Seong, David Hyunchul Shim

Abstract

Navigating unknown environments without prior maps remains a significant challenge for autonomous robots, often leading to local minima in complex topologies. We present the Skill Q-Network (SQN), a reinforcement learning architecture that learns an adaptive ensemble of latent skills. SQN decomposes navigation into perception, decision-making, and specialized skill execution. By incorporating a reachability-aware reward function, the model effectively balances goal-directed behavior with environment exploration. Our results demonstrate that SQN achieves a 98% success rate in maze environments and exhibits strong zero-shot generalization to unseen, large-scale subterranean caves, outperforming traditional monolithic RL baselines.

Introduction

Mapless navigation requires a robot to reach a target using only local observations, such as 2D range measurements. Standard Reinforcement Learning (RL) approaches typically employ a single, monolithic policy network. However, in environments with complex obstacles and dead ends, these models often fail to transition between exploring the space and navigating toward the goal. While Hierarchical RL (HRL) offers a potential solution, it frequently relies on manually defined sub-tasks. SQN addresses these limitations by discovering latent skills and learning an adaptive selection mechanism end-to-end.

Methodology

The SQN architecture is composed of four primary components designed to facilitate modular learning and adaptive behavior.

Detailed architecture of the Skill Q-Network (SQN) showing the Perception, Decision, and Skill modules.
  1. Perception Module: This module processes 2D range measurements and relative goal coordinates using Multi-Layer Perceptrons (MLP). A Gated Recurrent Unit (GRU) is integrated to provide the agent with temporal memory, essential for navigating partially observable environments.
  2. Decision Module: Acting as a high-level manager, this module outputs a "skill decision" ($\eta_t^\pi$)—an attention score that determines the relative importance of each latent skill for the current state.
  3. Latent Skill Policies: Instead of a single output head, SQN utilizes $N$ parallel skill heads. Each head learns a distinct navigational strategy (e.g., wall-following or open-space traversal) through experience.
  4. Skill Ensemble: The final action is determined by aggregating the Q-values from all latent skills, weighted by the attention scores from the Decision Module.

Reachability-Aware Reward

A key innovation in SQN is the "Reachability-Aware" reward function. Traditional rewards based solely on goal distance can lead to "optimism bias," where a robot attempts to move through walls to reach a nearby goal. SQN provides a goal-seeking reward only if the target is within the robot's current line-of-sight (reachable).

The concept of reachability: the goal is considered reachable only if it is within the robot's line-of-sight.

When the goal is blocked, the reward function prioritizes an exploration term, incentivizing the robot to map new areas and find a path around obstacles.

Experimental Results

The model was trained in a 17x17m maze and evaluated across several Out-of-Distribution (OOD) environments, including "Column," "Forest," and a large-scale "SubT Cave."

  • Success Rate and Efficiency: In the training maze, SQN achieved a 98% success rate, significantly higher than DQN and R2D2 baselines (~72%). Furthermore, SQN reached the goal using 42% fewer time steps on average.
  • Zero-Shot Generalization: SQN maintained an 82% success rate in the complex SubT Cave environment (51x51m), whereas the DQN baseline's performance dropped to 38%. This suggests that the latent skills learned are generalizable strategies rather than environment-specific paths.
  • Robustness: The architecture demonstrated high tolerance to sensor noise, maintaining over 80% success even with 60% uniform noise added to range measurements.

Conclusion

The Skill Q-Network demonstrates that functional modularity and reachability-aware rewards can significantly improve mapless navigation. By learning latent skills end-to-end, SQN avoids the need for manual task decomposition and achieves superior robustness in unseen environments. Future work will focus on bridging the sim-to-real gap and exploring dynamic skill allocation to further enhance navigational flexibility.

Summary updated

BibTeX
@inproceedings{seong2024skill,
  author = {Hyunki Seong and David Hyunchul Shim},
  title = {{Skill Q-Network: Learning Adaptive Skill Ensemble for Mapless Navigation in Unknown Environments}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2024},
  doi = {10.1109/iros58592.2024.10801531},
}
Figure