VER: Scaling On-Policy RL Leads to the Emergence of Navigation in Embodied Rearrangement

Erik Wijmans, Irfan Essa, Dhruv Batra

In the world of Embodied AI, scaling is everything. Training a robot to navigate a home or rearrange furniture often requires billions of steps of experience. However, a major bottleneck exists: some simulation environments are just slower than others. Whether it's complex physics or heavy rendering, these "stragglers" slow down the entire training process.

In their NeurIPS 2022 paper, researchers from Georgia Tech, Meta AI, and Google Atlanta introduce Variable Experience Rollout (VER). VER isn't just a speed boost; it’s a fundamental rethink of how we collect data for reinforcement learning (RL), leading to surprising "emergent" behaviors in robots.

TL;DR

VER is a new technique for scaling on-policy reinforcement learning that eliminates the "straggler effect" by allowing faster simulation environments to contribute more data than slower ones. It achieves up to a 2.7x speedup over state-of-the-art methods while maintaining high sample efficiency, eventually leading to the spontaneous emergence of navigation skills in robots trained only to pick up objects.

The "Straggler" Problem in Distributed RL

Most distributed RL systems fall into two camps:

  1. Synchronous (SyncOnRL): Every environment in a batch must finish its steps before the model updates. If one environment is slow (a "straggler"), the entire GPU sits idle.
  2. Asynchronous (AsyncOnRL): Environments update whenever they are ready. This is fast but "off-policy," meaning the data used for training is slightly stale, which hurts learning efficiency.
Comparison of SyncOnRL, VER, and AsyncOnRL timelines

The challenge is the heterogeneity of 3D environments. In a simulator like Habitat, rendering a complex kitchen takes longer than a hallway, and simulating a robot arm colliding with a fridge is computationally heavier than moving through empty space. VER aims to combine the high throughput of asynchronous methods with the superior sample efficiency of synchronous on-policy learning.

Variable Experience Rollout (VER)

The core innovation is Variable Experience Rollout (VER). Instead of forcing every environment to collect exactly $T$ steps, VER collects a total of $T \times N$ steps across $N$ environments, but allows the distribution to be uneven.

Variable Experience Rollout and Packed Sequence Format
  • Action-Level Mitigation: Using an architecture of "Environment Workers" and "Inference Workers," VER uses dynamic batching. It processes whatever environment requests are ready rather than waiting for a fixed batch size.
  • Environment-Level Mitigation: This is the "Variable" part. Fast environments might contribute 10 steps while a slow one contributes only 2.
  • PackedSequence Format: To handle these variable-length sequences on a GPU without wasting memory, the authors utilize a "packed" data format (similar to those used in NLP for variable-length sentences). This ensures the GPU stays busy regardless of how much data each environment provided.
  • Optimal Preemption: When scaling to multiple GPUs, VER calculates the optimal time to "cut off" slow workers and start the learning phase to maximize overall steps-per-second.

Key Insights: Emergent Navigation

The authors evaluated VER across several high-fidelity 3D simulation benchmarks, including Habitat 1.0, Habitat 2.0, and the Home Assistant Benchmark (HAB) using the ReplicaCAD dataset.

Beyond raw speed, the most striking finding was related to the handoff problem in hierarchical RL. When one skill (like "Navigate") hands off to another (like "Pick"), small errors accumulate. If the "Navigate" skill stops 5cm too far away, the "Pick" skill fails.

However, when "Pick" and "Place" skills were trained with the ability to move their base (even if they were spawned right next to the target), they spontaneously learned to navigate. If the robot missed a grasp and knocked an object away, it learned to drive after it. This emergent navigation made the robots significantly more robust to handoff errors.

Key Results

  • Speed: VER is 60-100% faster (1.6x-2x) than DD-PPO on standard navigation tasks. On complex mobile manipulation, it is 150-170% faster (up to 2.7x speedup).
  • Efficiency: Compared to SampleFactory (a leading asynchronous method), VER is 70% faster on 8 GPUs and demonstrates better sample efficiency because it stays strictly on-policy.
  • Performance: In the "Tidy House" task, using skills with emergent navigation improved the overall success rate from 2% to 32%—a massive jump for such a complex multi-step rearrangement problem.

Limitations and Future Work

While VER represents a significant leap in training efficiency, challenges remain:

  • Dataset Bias: The ReplicaCAD dataset is currently limited to layouts typical of US apartments, which may not generalize globally.
  • Oracle Knowledge: The current Task Planner used to chain skills still requires some "oracle" knowledge (e.g., knowing that a fridge must be opened before an item can be picked from it).
  • Hardware Contention: While VER optimizes GPU usage, as simulators become even more complex, the contention between rendering and learning on a single GPU will remain a hurdle that might eventually require more sophisticated multi-node scheduling.

For those looking to scale their own RL workloads, the code is available at the Facebook Research Habitat-Lab repository.

Summary updated

Abstract

We present Variable Experience Rollout (VER), a technique for efficiently scaling batched on-policy reinforcement learning in heterogenous environments (where different environments take vastly different times to generate rollouts) to many GPUs residing on, potentially, many machines. VER combines the strengths of and blurs the line between synchronous and asynchronous on-policy RL methods (SyncOnRL and AsyncOnRL, respectively). Specifically, it learns from on-policy experience (like SyncOnRL) and has no synchronization points (like AsyncOnRL) enabling high throughput.We find that VER leads to significant and consistent speed-ups across a broad range of embodied navigation and mobile manipulation tasks in photorealistic 3D simulation environments. Specifically, for PointGoal navigation and ObjectGoal navigation in Habitat 1.0, VER is 60-100% faster (1.6-2x speedup) than DD-PPO, the current state of art for distributed SyncOnRL, with similar sample efficiency. For mobile manipulation tasks (open fridge/cabinet, pick/place objects) in Habitat 2.0 VER is 150% faster (2.5x speedup) on 1 GPU and 170% faster (2.7x speedup) on 8 GPUs than DD-PPO. Compared to SampleFactory (the current state-of-the-art AsyncOnRL), VER matches its speed on 1 GPU, and is 70% faster (1.7x speedup) on 8 GPUs with better sample efficiency.We leverage these speed-ups to train chained skills for GeometricGoal rearrangement tasks in the Home Assistant Benchmark (HAB). We find a surprising emergence of navigation in skills that do not ostensible require any navigation. Specifically, the Pick skill involves a robot picking an object from a table. During training the robot was always spawned close to the table and never needed to navigate. However, we find that if base movement is part of the action space, the robot learns to navigate then pick an object in new environments with 50% success, demonstrating surprisingly high out-of-distribution generalization.
BibTeX
@inproceedings{wijmans2022ver,
  author = {Erik Wijmans and Irfan Essa and Dhruv Batra},
  title = {{VER: Scaling On-Policy RL Leads to the Emergence of Navigation in Embodied Rearrangement}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=vOBQDEMelt},
}
Figure