VER: Scaling On-Policy RL Leads to the Emergence of Navigation in Embodied Rearrangement

Erik Wijmans, Irfan Essa, Dhruv Batra

Abstract

We present Variable Experience Rollout (VER), a technique for efficiently scaling batched on-policy reinforcement learning in heterogenous environments to many GPUs. VER combines the strengths of synchronous and asynchronous on-policy RL methods by learning from on-policy experience without synchronization points. VER leads to significant speed-ups across embodied navigation and mobile manipulation tasks. Specifically, for PointGoal and ObjectGoal navigation in Habitat 1.0, VER is 1.6-2x faster than DD-PPO. For mobile manipulation in Habitat 2.0, VER is up to 2.7x faster. We leverage these speed-ups to train chained skills for rearrangement tasks and find a surprising emergence of navigation in skills that do not ostensibly require it, demonstrating high out-of-distribution generalization.

Introduction

Training embodied AI agents—robots that perceive and interact with 3D environments—requires billions of steps of experience. Traditionally, scaling this process involves a trade-off between Synchronous RL (SyncOnRL), which suffers from the "straggler effect" where idle GPUs wait for slow environment simulations, and Asynchronous RL (AsyncOnRL), which achieves high throughput but suffers from reduced sample efficiency due to "stale" data.

In high-fidelity 3D simulations like Habitat, the straggler effect is particularly severe because simulation time is heterogeneous; for instance, simulating a collision with a complex object takes much longer than a simple movement. VER addresses this by bridging the gap between these two paradigms.

Methodology: Variable Experience Rollout (VER)

The core innovation of this work is Variable Experience Rollout (VER). Unlike traditional methods that force every environment to produce a fixed number of steps, VER allows faster environments to contribute more steps and slower ones to contribute fewer.

Comparison of VER with traditional Synchronous and Asynchronous RL methods.

Key technical features include:

  • Elimination of Synchronization Points: VER computes actions and updates the policy as soon as data is available, ensuring the GPU remains fully utilized.
  • Variable Rollout Lengths: By explicitly collecting more experience from fast-to-simulate environments, VER maximizes throughput in heterogeneous settings.
  • Efficient Data Handling: To manage variable-length sequences for recurrent neural networks, the system utilizes a PackedSequence data format to maintain memory contiguity and computational efficiency.

Experimental Evaluation

VER was evaluated across several rigorous 3D benchmarks:

  • Habitat 1.0: PointGoal and ObjectGoal navigation using the HM3D and MP3D datasets.
  • Habitat 2.0 & ReplicaCAD: Complex mobile manipulation tasks such as opening cabinets and picking/placing objects.
  • Home Assistant Benchmark (HAB): Multi-stage rearrangement tasks like Tidy House, Prepare Groceries, and Set Table.

Results and Emergent Behavior

VER demonstrated consistent and substantial speedups over the current state-of-the-art, DD-PPO. In navigation tasks, VER was 60-100% faster, while in mobile manipulation, it achieved 150-170% faster training. Crucially, this speed was achieved with similar or better sample efficiency than previous methods.

Performance benchmarks for PointGoal and ObjectGoal navigation, highlighting significant speedups.

Perhaps the most striking result is the emergence of navigation. When training a "Pick" skill where the robot was typically spawned near an object, the researchers included base movement in the action space. Because VER allowed for much more extensive training, the robot spontaneously learned to navigate across rooms to reach objects from entirely new starting positions—a behavior it was never explicitly taught. This emergent capability improved success rates on the complex "Tidy House" benchmark from 2% to 32%.

Conclusion and Future Work

VER represents a significant step forward in scaling embodied AI training by effectively neutralizing the straggler effect in complex simulations. While highly effective, the authors note that future work should address the limited diversity of current 3D datasets and reduce the reliance on "oracle knowledge" for high-level task planning. Further optimizations to overlap the learning and collection phases could potentially push throughput even higher.

Summary updated

Abstract

We present Variable Experience Rollout (VER), a technique for efficiently scaling batched on-policy reinforcement learning in heterogenous environments (where different environments take vastly different times to generate rollouts) to many GPUs residing on, potentially, many machines. VER combines the strengths of and blurs the line between synchronous and asynchronous on-policy RL methods (SyncOnRL and AsyncOnRL, respectively). VER learns from on-policy experience (like SyncOnRL) and has no synchronization points (like AsyncOnRL). VER leads to significant and consistent speed-ups across a broad range of embodied navigation and mobile manipulation tasks in photorealistic 3D simulation environments. Specifically, for PointGoal navigation and ObjectGoal navigation in Habitat 1.0, VER is 60-100% faster (1.6-2x speedup) than DD-PPO, the current state of art distributed SyncOnRL, with similar sample efficiency. For mobile manipulation tasks (open fridge/cabinet, pick/place objects) in Habitat 2.0 VER is 150% faster (2.5x speedup) on 1 GPU and 170% faster (2.7x speedup) on 8 GPUs than DD-PPO. Compared to SampleFactory (the current state-of-the-art AsyncOnRL), VER matches its speed on 1 GPU, and is 70% faster (1.7x speedup) on 8 GPUs with better sample efficiency. We leverage these speed-ups to train chained skills for GeometricGoal rearrangement tasks in the Home Assistant Benchmark (HAB). We find a surprising emergence of navigation in skills that do not ostensible require any navigation. Specifically, the Pick skill involves a robot picking an object from a table. During training the robot was always spawned close to the table and never needed to navigate. However, we find that if base movement is part of the action space, the robot learns to navigate then pick an object in new environments with 50% success, demonstrating surprisingly high out-of-distribution generalization.
BibTeX
@inproceedings{wijmans2022ver,
  author = {Erik Wijmans and Irfan Essa and Dhruv Batra},
  title = {{VER: Scaling On-Policy RL Leads to the Emergence of Navigation in Embodied Rearrangement}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=vOBQDEMelt},
}
Figure