Abstract
We present Variable Experience Rollout (VER), a technique for efficiently scaling batched on-policy reinforcement learning in heterogenous environments to many GPUs. VER combines the strengths of synchronous and asynchronous on-policy RL methods by learning from on-policy experience without synchronization points. VER leads to significant speed-ups across embodied navigation and mobile manipulation tasks. Specifically, for PointGoal and ObjectGoal navigation in Habitat 1.0, VER is 1.6-2x faster than DD-PPO. For mobile manipulation in Habitat 2.0, VER is up to 2.7x faster. We leverage these speed-ups to train chained skills for rearrangement tasks and find a surprising emergence of navigation in skills that do not ostensibly require it, demonstrating high out-of-distribution generalization.
Introduction
Training embodied AI agents—robots that perceive and interact with 3D environments—requires billions of steps of experience. Traditionally, scaling this process involves a trade-off between Synchronous RL (SyncOnRL), which suffers from the "straggler effect" where idle GPUs wait for slow environment simulations, and Asynchronous RL (AsyncOnRL), which achieves high throughput but suffers from reduced sample efficiency due to "stale" data.
In high-fidelity 3D simulations like Habitat, the straggler effect is particularly severe because simulation time is heterogeneous; for instance, simulating a collision with a complex object takes much longer than a simple movement. VER addresses this by bridging the gap between these two paradigms.
Methodology: Variable Experience Rollout (VER)
The core innovation of this work is Variable Experience Rollout (VER). Unlike traditional methods that force every environment to produce a fixed number of steps, VER allows faster environments to contribute more steps and slower ones to contribute fewer.
Key technical features include:
- Elimination of Synchronization Points: VER computes actions and updates the policy as soon as data is available, ensuring the GPU remains fully utilized.
- Variable Rollout Lengths: By explicitly collecting more experience from fast-to-simulate environments, VER maximizes throughput in heterogeneous settings.
- Efficient Data Handling: To manage variable-length sequences for recurrent neural networks, the system utilizes a
PackedSequencedata format to maintain memory contiguity and computational efficiency.
Experimental Evaluation
VER was evaluated across several rigorous 3D benchmarks:
- Habitat 1.0: PointGoal and ObjectGoal navigation using the HM3D and MP3D datasets.
- Habitat 2.0 & ReplicaCAD: Complex mobile manipulation tasks such as opening cabinets and picking/placing objects.
- Home Assistant Benchmark (HAB): Multi-stage rearrangement tasks like Tidy House, Prepare Groceries, and Set Table.
Results and Emergent Behavior
VER demonstrated consistent and substantial speedups over the current state-of-the-art, DD-PPO. In navigation tasks, VER was 60-100% faster, while in mobile manipulation, it achieved 150-170% faster training. Crucially, this speed was achieved with similar or better sample efficiency than previous methods.
Perhaps the most striking result is the emergence of navigation. When training a "Pick" skill where the robot was typically spawned near an object, the researchers included base movement in the action space. Because VER allowed for much more extensive training, the robot spontaneously learned to navigate across rooms to reach objects from entirely new starting positions—a behavior it was never explicitly taught. This emergent capability improved success rates on the complex "Tidy House" benchmark from 2% to 32%.
Conclusion and Future Work
VER represents a significant step forward in scaling embodied AI training by effectively neutralizing the straggler effect in complex simulations. While highly effective, the authors note that future work should address the limited diversity of current 3D datasets and reduce the reliance on "oracle knowledge" for high-level task planning. Further optimizations to overlap the learning and collection phases could potentially push throughput even higher.
Summary updated