In the world of Embodied AI, scaling is everything. Training a robot to navigate a home or rearrange furniture often requires billions of steps of experience. However, a major bottleneck exists: some simulation environments are just slower than others. Whether it's complex physics or heavy rendering, these "stragglers" slow down the entire training process.
In their NeurIPS 2022 paper, researchers from Georgia Tech, Meta AI, and Google Atlanta introduce Variable Experience Rollout (VER). VER isn't just a speed boost; it’s a fundamental rethink of how we collect data for reinforcement learning (RL), leading to surprising "emergent" behaviors in robots.
TL;DR
VER is a new technique for scaling on-policy reinforcement learning that eliminates the "straggler effect" by allowing faster simulation environments to contribute more data than slower ones. It achieves up to a 2.7x speedup over state-of-the-art methods while maintaining high sample efficiency, eventually leading to the spontaneous emergence of navigation skills in robots trained only to pick up objects.
The "Straggler" Problem in Distributed RL
Most distributed RL systems fall into two camps:
- Synchronous (SyncOnRL): Every environment in a batch must finish its steps before the model updates. If one environment is slow (a "straggler"), the entire GPU sits idle.
- Asynchronous (AsyncOnRL): Environments update whenever they are ready. This is fast but "off-policy," meaning the data used for training is slightly stale, which hurts learning efficiency.
The challenge is the heterogeneity of 3D environments. In a simulator like Habitat, rendering a complex kitchen takes longer than a hallway, and simulating a robot arm colliding with a fridge is computationally heavier than moving through empty space. VER aims to combine the high throughput of asynchronous methods with the superior sample efficiency of synchronous on-policy learning.
Variable Experience Rollout (VER)
The core innovation is Variable Experience Rollout (VER). Instead of forcing every environment to collect exactly $T$ steps, VER collects a total of $T \times N$ steps across $N$ environments, but allows the distribution to be uneven.
- Action-Level Mitigation: Using an architecture of "Environment Workers" and "Inference Workers," VER uses dynamic batching. It processes whatever environment requests are ready rather than waiting for a fixed batch size.
- Environment-Level Mitigation: This is the "Variable" part. Fast environments might contribute 10 steps while a slow one contributes only 2.
- PackedSequence Format: To handle these variable-length sequences on a GPU without wasting memory, the authors utilize a "packed" data format (similar to those used in NLP for variable-length sentences). This ensures the GPU stays busy regardless of how much data each environment provided.
- Optimal Preemption: When scaling to multiple GPUs, VER calculates the optimal time to "cut off" slow workers and start the learning phase to maximize overall steps-per-second.
Key Insights: Emergent Navigation
The authors evaluated VER across several high-fidelity 3D simulation benchmarks, including Habitat 1.0, Habitat 2.0, and the Home Assistant Benchmark (HAB) using the ReplicaCAD dataset.
Beyond raw speed, the most striking finding was related to the handoff problem in hierarchical RL. When one skill (like "Navigate") hands off to another (like "Pick"), small errors accumulate. If the "Navigate" skill stops 5cm too far away, the "Pick" skill fails.
However, when "Pick" and "Place" skills were trained with the ability to move their base (even if they were spawned right next to the target), they spontaneously learned to navigate. If the robot missed a grasp and knocked an object away, it learned to drive after it. This emergent navigation made the robots significantly more robust to handoff errors.
Key Results
- Speed: VER is 60-100% faster (1.6x-2x) than DD-PPO on standard navigation tasks. On complex mobile manipulation, it is 150-170% faster (up to 2.7x speedup).
- Efficiency: Compared to SampleFactory (a leading asynchronous method), VER is 70% faster on 8 GPUs and demonstrates better sample efficiency because it stays strictly on-policy.
- Performance: In the "Tidy House" task, using skills with emergent navigation improved the overall success rate from 2% to 32%—a massive jump for such a complex multi-step rearrangement problem.
Limitations and Future Work
While VER represents a significant leap in training efficiency, challenges remain:
- Dataset Bias: The ReplicaCAD dataset is currently limited to layouts typical of US apartments, which may not generalize globally.
- Oracle Knowledge: The current Task Planner used to chain skills still requires some "oracle" knowledge (e.g., knowing that a fridge must be opened before an item can be picked from it).
- Hardware Contention: While VER optimizes GPU usage, as simulators become even more complex, the contention between rendering and learning on a single GPU will remain a hurdle that might eventually require more sophisticated multi-node scheduling.
For those looking to scale their own RL workloads, the code is available at the Facebook Research Habitat-Lab repository.
Summary updated