Curiosity-driven Exploration for Mapless Navigation with Deep Reinforcement Learning

Oleksii Zhelo, Jingwei Zhang, Lei Tai, Ming Liu, Wolfram BurgardarXiv 2018 · OpenReview

Abstract

This paper investigates exploration strategies of Deep Reinforcement Learning (DRL) methods to learn navigation policies for mobile robots. In particular, we augment the normal external reward for training DRL algorithms with intrinsic reward signals measured by curiosity. We test our approach in a mapless navigation setting, where the autonomous agent is required to navigate without the occupancy map of the environment, to targets whose relative locations can be easily acquired through low-cost solutions (e.g., visible light localization, Wi-Fi signal localization). We validate that the intrinsic motivation is crucial for improving DRL performance in tasks with challenging exploration requirements. Our experimental results show that our proposed method is able to more effectively learn navigation policies, and has better generalization capabilities in previously unseen environments.

Introduction

Traditional robot navigation often relies on detailed maps created through Simultaneous Localization and Mapping (SLAM). However, mapless navigation offers a more flexible alternative, where a robot learns to navigate using only local sensor data—such as 2D laser scans—and a relative target position.

The primary hurdle in training such systems via Deep Reinforcement Learning (DRL) is the "sparse reward" problem. In complex environments with dead ends and long corridors, a robot might rarely reach the goal by chance, leading to stalled learning. This paper introduces a curiosity-driven approach that augments standard DRL with an Intrinsic Curiosity Module (ICM) to encourage active exploration of the environment.

Methodology

The proposed system is built upon the Asynchronous Advantage Actor-Critic (A3C) algorithm, enhanced by the ICM. The ICM generates an "intrinsic reward" based on the robot's ability to predict the consequences of its actions.

The Intrinsic Curiosity Module (ICM)

The ICM consists of two primary components that work together to generate a curiosity signal:

  1. The Inverse Model ($\psi^i$): Learns to predict the action $a_t$ taken to transition from state $s_t$ to $s_{t+1}$. This encourages the network to focus on features that the robot can control, ignoring environmental "noise."
  2. The Forward Model ($\psi^f$): Predicts the feature representation of the next state $\hat{\phi}_{t+1}$ given the current state and action.

The prediction error of the forward model—the difference between the predicted and actual next state—serves as the intrinsic reward. When the robot encounters unfamiliar territory, the prediction error is high, providing a "curiosity" signal that rewards exploration.

Architecture of the Intrinsic Curiosity Module (ICM) showing the inverse and forward models.

Experimental Setup

The robot's input consists of 72-dimensional laser range readings and the relative goal position. It outputs discrete actions: moving straight, turning left, or turning right. The architecture utilizes Convolutional Neural Networks (CNNs) for sensor processing and Long Short-Term Memory (LSTM) layers to provide temporal context.

The approach was evaluated across four simulated 2D floor plans:

  • Map 1: The primary training environment.
  • Map 2: Includes structural variations like angled walls.
  • Maps 3 & 4: Increasingly complex layouts used to test the policy's generalization to unseen environments.

Results and Discussion

The researchers compared four strategies: a baseline (A3C-), Entropy-only exploration, ICM-only, and a combined ICM + Entropy approach.

Performance and Efficiency

On the training map, the ICM + Entropy model achieved a 100% success rate, significantly outperforming the baseline's 88.3%. Furthermore, curious agents were more efficient, reaching the goal in an average of 75 steps compared to the baseline's 173 steps.

Training performance comparison across average reward and average steps.

Generalization and Stability

The curiosity-driven agent demonstrated superior generalization. In the most challenging unseen environment (Map 4), the ICM + Entropy model maintained a 55% success rate, while the Entropy-only approach dropped to 34.3%. Training curves indicate that curiosity provides a more stable learning signal, helping the agent escape local minima and avoid sub-optimal behaviors like taking excessively long, "safe" routes.

Conclusion

This work demonstrates that intrinsic motivation is a powerful tool for mapless navigation. By seeking out unpredictable states, robots can learn more robust policies that generalize better to new environments. Future work will focus on transitioning these models from simulation to real-world hardware, addressing sensor noise and physical constraints, and automating the balance between intrinsic and extrinsic rewards.

A video of the experimental results is available at https://goo.gl/pWbpcF.

Summary updated

Abstract

This paper investigates exploration strategies of Deep Reinforcement Learning (DRL) methods to learn navigation policies for mobile robots. In particular, we augment the normal external reward for training DRL algorithms with intrinsic reward signals measured by curiosity. We test our approach in a mapless navigation setting, where the autonomous agent is required to navigate without the occupancy map of the environment, to targets whose relative locations can be easily acquired through low-cost solutions (e.g., visible light localization, Wi-Fi signal localization). We validate that the intrinsic motivation is crucial for improving DRL performance in tasks with challenging exploration requirements. Our experimental results show that our proposed method is able to more effectively learn navigation policies, and has better generalization capabilities in previously unseen environments. A video of our experimental results can be found at https://goo.gl/pWbpcF.
BibTeX
@misc{zhelo2018curiosity,
  author = {Oleksii Zhelo and Jingwei Zhang and Lei Tai and Ming Liu and Wolfram Burgard},
  title = {{Curiosity-driven Exploration for Mapless Navigation with Deep Reinforcement Learning}},
  year = {2018},
  eprint = {1804.00456},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1804.00456},
}
Figure