Abstract: Numerous past works have tackled the problem of task-driven navigation. But, how to effectively explore a new environment to enable a variety of down-stream tasks has received much less attention. In this work, we study how agents can autonomously explore realistic and complex 3D environments without the context of task-rewards. We propose a learning-based approach and investigate different policy architectures, reward functions, and training paradigms. We find that the use of policies with spatial memory that are bootstrapped with imitation learning and finally finetuned with coverage rewards derived purely from on-board sensors can be effective at exploring novel environments. We show that our learned exploration policies can explore better than classical approaches based on geometry alone and generic learning-based exploration techniques. Finally, we also show how such task-agnostic exploration can be used for down-stream tasks.
Introduction
Most research in robotic navigation focuses on task-driven behavior—getting from point A to point B. However, these methods often assume the agent already has a map or can rely on a human to build one. In the real world, an agent’s first job is often just to "look around" and get its bearings.
Current solutions typically fall into two camps:
- Classical Geometric Approaches: These are effective but brittle, failing under sensor noise or when semantic understanding (e.g., recognizing a door as a traversable path) is required.
- Learning-based Approaches: Methods using "curiosity" or random exploration are often sample-inefficient and struggle with long-term consistency.
This paper bridges this gap by learning a policy that explores efficiently and builds a representation useful for any future task.
Methodology
The authors propose a hybrid architecture that fuses classical spatial reasoning with deep learning. The agent constructs a 2D occupancy map on the fly using depth sensors to track traversable space.
The policy architecture utilizes a Recurrent Neural Network (GRU) that processes three primary inputs: RGB images for semantic cues, and two-scale egocentric maps (a "fine" map for local obstacles and a "coarse" map for long-range planning). Instead of task-specific rewards, the agent is trained using an Intrinsic Coverage Reward, gaining points for every new square meter explored while being penalized for collisions. Training follows a two-step paradigm: Imitation Learning (IL) to mimic human exploration trajectories, followed by Reinforcement Learning (RL) using Proximal Policy Optimization (PPO) to maximize coverage.
Experimental Setup
Experiments were conducted in the House3D simulation environment, utilizing realistic 3D apartment layouts from the SUNCG dataset. The model was trained on 20 houses and evaluated on 20 separate, held-out houses to ensure generalization. Human data for the imitation learning phase consisted of 693 exploration trajectories collected via Amazon Mechanical Turk.
Results and Analysis
The learned policy demonstrates significant advantages over both classical and generic learning-based baselines in terms of efficiency, robustness, and semantic awareness.
Key Findings:
- Exploration Efficiency: The full system (IL + RL with Map + RGB) significantly outperformed curiosity-based RL and random baselines in area coverage over 1,000 time steps.
- Semantic Awareness: Unlike geometric planners, the learned policy "understands" that a door is a gateway to more coverage. In experiments where doors were rendered but not marked as "free space" in the geometry, the learned agent successfully navigated through them while classical methods failed.
- Robustness to Noise: Geometric methods like Frontier-based exploration collapse when sensor noise is introduced. The learned policy remains robust, outperforming the classical method at noise levels as low as 4%.
- Downstream Utility: Maps built during this task-agnostic phase significantly improve performance on future navigation tasks, increasing Success weighted by Path Length (SPL) from 0.69 to 0.87.
Conclusion
This work demonstrates that combining spatial memory, imitation learning, and intrinsic coverage rewards creates effective exploration policies for complex 3D environments. While currently limited by 2D projections and simplified noise models, the approach paves the way for "semantic exploration" where agents can prioritize specific areas based on high-level cues. Future work will focus on closing the sim-to-real gap and handling more complex 3D structures.
Summary updated