TL;DR
Cyclic Policy Distillation (CPD) is a new reinforcement learning framework that makes "Domain Randomization" significantly more sample-efficient. By breaking a wide range of simulation parameters into smaller sub-domains and cyclically transferring knowledge between them, CPD achieves up to 6x better sample efficiency than existing methods and enables successful zero-shot transfer to complex real-robot tasks.
What Problem Does This Paper Address?
To get a robot to work in the real world, researchers often use Domain Randomization (DR). The idea is simple: train the robot in a simulation where physical properties (like friction, mass, or gravity) change constantly. If the robot learns to succeed across all these variations, it should be robust enough to handle the "reality gap" when deployed on a physical machine.
However, DR has a major "brute force" problem. When the range of randomized parameters is large, the learning process becomes unstable. The policy gets "confused" by the massive variance in experiences, requiring millions of samples to converge. Existing fixes—like curriculum learning or dividing the task into independent sub-problems—often require manual tuning or fail to share useful knowledge between similar environments.
Methods
The authors introduce Cyclic Policy Distillation (CPD), which operates in three main stages:
- Domain Partitioning: Instead of training one policy on the entire range of randomized parameters, CPD divides the range into several overlapping sub-domains (e.g., one sub-domain for low friction, one for medium, one for high).
- Cyclic Learning with Knowledge Transfer: This is the core innovation. The agent trains on one sub-domain at a time, moving cyclically from sub-domain $1$ to $N$ and then back again. Crucially, when moving to a new sub-domain, the agent doesn't start from scratch. It uses a Monotonic Policy Improvement scheme to "mix" its current policy with the knowledge of its neighbor. This ensures that updates are stable and that the agent leverages similarities between adjacent environments.
- Global Distillation: Once all local policies are experts in their specific sub-domains, they are distilled (merged) into a single global policy capable of handling the entire range for real-world deployment.
Datasets
The researchers evaluated CPD across several benchmarks:
- OpenAI Gym: The classic Pendulum task.
- MuJoCo Simulations: Three complex continuous control tasks: Pusher, Swimmer, and HalfCheetah.
- Real-Robot Task: A "ball-dispersal" task using a UR3 robot arm. The goal is to knock down a pile of balls until they are all flat on a surface. This is particularly challenging because it involves complex multi-object interactions and noisy depth-camera data.
Key Insights and Ideas
- Neighborly Advice: The "aha moment" here is that neighboring sub-domains (e.g., mass=1.0kg vs mass=1.1kg) are mathematically similar. CPD exploits this by using the value function of a previous sub-domain to jump-start learning in the next.
- Smart Mixing vs. Naive Copying: The paper shows that simply copying a policy from one domain to another can actually hurt performance due to the "domain gap." CPD uses a dynamic "mixture rate" ($m$) based on expected performance improvement, ensuring the agent only listens to its neighbor when it's actually helpful.
- Stability via Cycles: By transitioning cyclically (1 $\to$ 2 $\to$ 3 $\to$ 2 $\to$ 1), the agent maintains expertise across the whole spectrum rather than forgetting old skills as it learns new ones.
Key Results
- Efficiency Gains: In the Pendulum task, CPD achieved 3x to 6x better sample efficiency compared to state-of-the-art methods like P2PDRL or Divide-and-Conquer (DnC).
- Higher Performance Ceiling: In the MuJoCo HalfCheetah task, CPD was the only method that learned an effective policy within a practical number of samples, while standard Domain Randomization (SAC-DR) barely got off the ground.
- Real-World Success: In the ball-dispersal task, the CPD-trained policy achieved a 40% higher reward than baselines. It successfully transferred to the real robot in a zero-shot setting, even when tested with different types of balls (hard juggling balls vs. soft, deformable beads) that it had never seen in simulation.
Limitations and Future Work
- Manual Partitioning: Currently, a human designer has to decide which parameter to split (e.g., mass vs. friction) and how many divisions ($N$) to create. The authors note that $N$ involves a trade-off: too few divisions lead to instability, while too many can slow down the overall distillation.
- Parallelization: Because the method is "cyclic" (sequential), it isn't immediately obvious how to run it on massive parallel compute clusters, though the authors suggest grouping sub-domains into clusters as a potential workaround.
- Theoretical Gap: While the method works empirically, the theoretical guarantees for "Monotonic Improvement" were originally designed for single domains; proving they hold across randomized multi-domains remains an open research question.
Summary updated