Learning Exploration Policies for Navigation

Tao Chen, Saurabh Gupta, Abhinav Gupta

Imagine walking into a new office for the first time. You don’t immediately run to a specific desk; instead, you wander around, noting where the kitchen is, identifying hallways, and figuring out which doors lead to meeting rooms. This "getting to know the place" phase is critical for humans, yet in the world of robotics, we usually skip it, forcing agents to learn navigation only in the context of a specific goal (like "find the mug").

In their ICLR 2019 paper, "Learning Exploration Policies for Navigation," Tao Chen and colleagues from CMU and Facebook AI Research argue that we need to treat exploration as a first-class citizen. They propose a way for robots to autonomously explore complex 3D environments to build a "mental map" that makes all future tasks easier.

The Exploration Blind Spot

Most navigation research falls into two camps: classical robotics, which uses geometric mapping (SLAM) and path planning, and learning-based methods, which train end-to-end policies for specific tasks.

Both have "exploration" blind spots. Classical methods often rely on "frontier-based exploration" (simply moving toward the edge of the known map), which is brittle when sensors are noisy or when the robot needs to understand semantic cues (like recognizing that a door is a traversable pathway). Meanwhile, learning-based agents are usually "goal-driven," meaning they only learn to move when there is a reward at the end of a specific path. If you drop them in a brand-new house without a goal, they often struggle to move efficiently.

This paper asks: How can we learn a policy that explores a novel environment efficiently to enable a variety of future downstream tasks?

A Hybrid Architecture for Task-Agnostic Exploration

The authors propose a hybrid architecture that fuses three distinct types of information to build a robust exploration policy:

  1. RGB Images: Processed via a ResNet-18 to identify semantic cues (like doors or hallways).
  2. Occupancy Maps: The robot maintains a running 2D map of where it has been. It looks at this map at two scales: a "fine" local map (4m x 4m) and a "coarse" global map (40m x 40m).
  3. Recurrent Memory: A GRU (Gated Recurrent Unit) helps the agent maintain temporal coherence, preventing it from just spinning in circles.
The proposed hybrid architecture combining RGB images, occupancy maps, and recurrent memory, trained via PPO with coverage rewards.

The Training Recipe

Training a robot to explore from scratch is hard because the "reward" is sparse. To solve this, the authors used a two-step process:

  • Imitation Learning (Bootstrapping): They first trained the agent to mimic human trajectories. Humans are naturally good at exploring, so this gave the agent a "warm start" on how to navigate through doors and around obstacles.
  • Reinforcement Learning (PPO): They then fine-tuned the agent using Proximal Policy Optimization with a Coverage Reward. The agent gets a "paycheck" every time it uncovers a new, previously unseen area on its occupancy map.

Key Insights and Results

The study was conducted using the House3D simulation environment, utilizing 20 realistic 3D house layouts for training and a separate set of 20 for testing. The environment was specifically modified to test "Geometry and Affordance Mismatch"—for example, rendering doors that the robot must learn are traversable, rather than treating them as permanent walls.

Exploration performance comparisons showing ablation results (left), robustness to sensor noise (middle), and performance in environments with doors (right).
  • Semantics Beat Pure Geometry: Classical frontier-based methods treat a door and a wall as the same thing: an obstacle. The learned policy, however, learns that a "door" shape in an RGB image is a high-probability pathway to more "coverage," allowing it to prioritize rooms more effectively.
  • Robustness to Noise: This is a critical "aha" moment. Classical mapping fails catastrophically when the robot’s self-localization is noisy (e.g., the wheels slip). Because the learned policy is trained on visual cues and maps simultaneously, it is much more resilient to these real-world imperfections. As soon as 4% noise is introduced to the robot's movement, the learned policy significantly outperforms the classical baseline.
  • Ablation Success: The "Full System" (Map + RGB + RL + Imitation) performed the best. Removing the map or the RGB input led to a noticeable drop in how much of the house the robot could see within 1,000 steps.
  • Downstream Utility: To prove the exploration was useful, they tested a "Goal-Directed Navigation" task. Agents that used the map built during the exploration phase achieved a Success weighted by Path Length (SPL) of 0.87, compared to only 0.69 for agents starting with no prior knowledge.

Limitations and Future Work

While the results are promising, the authors acknowledge a few hurdles. First, the work was done entirely in simulation; real-world physics and lighting might pose tougher challenges for RGB-based semantic recognition. Second, the agent uses "macro-actions" (e.g., move forward 0.25m, turn 9°), whereas real-world continuous control is much more complex.

Looking ahead, the authors suggest moving toward "Semantic Exploration"—where a robot doesn't just look for any empty space, but specifically looks for "the kitchen" or "the bedroom" based on high-level cues.

Reference: Chen, T., Gupta, S., & Gupta, A. (2019). Learning Exploration Policies for Navigation. Published as a conference paper at ICLR 2019.

Summary updated

Abstract

Numerous past works have tackled the problem of task-driven navigation. But, how to effectively explore a new environment to enable a variety of down-stream tasks has received much less attention. In this work, we study how agents can autonomously explore realistic and complex 3D environments without the context of task-rewards. We propose a learning-based approach and investigate different policy architectures, reward functions, and training paradigms. We find that use of policies with spatial memory that are bootstrapped with imitation learning and finally finetuned with coverage rewards derived purely from on-board sensors can be effective at exploring novel environments. We show that our learned exploration policies can explore better than classical approaches based on geometry alone and generic learning-based exploration techniques. Finally, we also show how such task-agnostic exploration can be used for down-stream tasks. Videos are available at https://sites.google.com/view/exploration-for-nav/.
BibTeX
@inproceedings{chen2019learning,
  author = {Tao Chen and Saurabh Gupta and Abhinav Gupta},
  title = {{Learning Exploration Policies for Navigation}},
  booktitle = {International Conference on Learning Representations},
  year = {2019},
  url = {https://openreview.net/forum?id=f97d51RPCJT7},
}
Figure