Cyclic policy distillation: Sample-efficient sim-to-real reinforcement learning with domain randomization

Yuki Kadokawa, Lingwei Zhu, Yoshihisa Tsurumine, Takamitsu Matsubara

TL;DR

Cyclic Policy Distillation (CPD) is a new reinforcement learning framework that makes "Domain Randomization" significantly more sample-efficient. By breaking a wide range of simulation parameters into smaller sub-domains and cyclically transferring knowledge between them, CPD achieves up to 6x better sample efficiency than existing methods and enables successful zero-shot transfer to complex real-robot tasks.

What Problem Does This Paper Address?

To get a robot to work in the real world, researchers often use Domain Randomization (DR). The idea is simple: train the robot in a simulation where physical properties (like friction, mass, or gravity) change constantly. If the robot learns to succeed across all these variations, it should be robust enough to handle the "reality gap" when deployed on a physical machine.

However, DR has a major "brute force" problem. When the range of randomized parameters is large, the learning process becomes unstable. The policy gets "confused" by the massive variance in experiences, requiring millions of samples to converge. Existing fixes—like curriculum learning or dividing the task into independent sub-problems—often require manual tuning or fail to share useful knowledge between similar environments.

Methods

The authors introduce Cyclic Policy Distillation (CPD), which operates in three main stages:

  1. Domain Partitioning: Instead of training one policy on the entire range of randomized parameters, CPD divides the range into several overlapping sub-domains (e.g., one sub-domain for low friction, one for medium, one for high).
  2. Cyclic Learning with Knowledge Transfer: This is the core innovation. The agent trains on one sub-domain at a time, moving cyclically from sub-domain $1$ to $N$ and then back again. Crucially, when moving to a new sub-domain, the agent doesn't start from scratch. It uses a Monotonic Policy Improvement scheme to "mix" its current policy with the knowledge of its neighbor. This ensures that updates are stable and that the agent leverages similarities between adjacent environments.
  3. Global Distillation: Once all local policies are experts in their specific sub-domains, they are distilled (merged) into a single global policy capable of handling the entire range for real-world deployment.

Datasets

The researchers evaluated CPD across several benchmarks:

  • OpenAI Gym: The classic Pendulum task.
  • MuJoCo Simulations: Three complex continuous control tasks: Pusher, Swimmer, and HalfCheetah.
  • Real-Robot Task: A "ball-dispersal" task using a UR3 robot arm. The goal is to knock down a pile of balls until they are all flat on a surface. This is particularly challenging because it involves complex multi-object interactions and noisy depth-camera data.

Key Insights and Ideas

  • Neighborly Advice: The "aha moment" here is that neighboring sub-domains (e.g., mass=1.0kg vs mass=1.1kg) are mathematically similar. CPD exploits this by using the value function of a previous sub-domain to jump-start learning in the next.
  • Smart Mixing vs. Naive Copying: The paper shows that simply copying a policy from one domain to another can actually hurt performance due to the "domain gap." CPD uses a dynamic "mixture rate" ($m$) based on expected performance improvement, ensuring the agent only listens to its neighbor when it's actually helpful.
  • Stability via Cycles: By transitioning cyclically (1 $\to$ 2 $\to$ 3 $\to$ 2 $\to$ 1), the agent maintains expertise across the whole spectrum rather than forgetting old skills as it learns new ones.

Key Results

  • Efficiency Gains: In the Pendulum task, CPD achieved 3x to 6x better sample efficiency compared to state-of-the-art methods like P2PDRL or Divide-and-Conquer (DnC).
  • Higher Performance Ceiling: In the MuJoCo HalfCheetah task, CPD was the only method that learned an effective policy within a practical number of samples, while standard Domain Randomization (SAC-DR) barely got off the ground.
  • Real-World Success: In the ball-dispersal task, the CPD-trained policy achieved a 40% higher reward than baselines. It successfully transferred to the real robot in a zero-shot setting, even when tested with different types of balls (hard juggling balls vs. soft, deformable beads) that it had never seen in simulation.

Limitations and Future Work

  • Manual Partitioning: Currently, a human designer has to decide which parameter to split (e.g., mass vs. friction) and how many divisions ($N$) to create. The authors note that $N$ involves a trade-off: too few divisions lead to instability, while too many can slow down the overall distillation.
  • Parallelization: Because the method is "cyclic" (sequential), it isn't immediately obvious how to run it on massive parallel compute clusters, though the authors suggest grouping sub-domains into clusters as a potential workaround.
  • Theoretical Gap: While the method works empirically, the theoretical guarantees for "Monotonic Improvement" were originally designed for single domains; proving they hold across randomized multi-domains remains an open research question.

Summary updated

Abstract

Deep reinforcement learning with domain randomization learns a control policy in various simulations with randomized physical and sensor model parameters to become transferable to the real world in a zero-shot setting. However, a huge number of samples are often required to learn an effective policy when the range of randomized parameters is extensive due to the instability of policy updates. To alleviate this problem, we propose a sample-efficient method named cyclic policy distillation (CPD). CPD divides the range of randomized parameters into several small sub-domains and assigns a local policy to each one. Then local policies are learned while cyclically transitioning to sub-domains. CPD accelerates learning through knowledge transfer based on expected performance improvements. Finally, all of the learned local policies are distilled into a global policy for sim-to-real transfers. CPD's effectiveness and sample efficiency are demonstrated through simulations with four tasks (Pendulum from OpenAIGym and Pusher, Swimmer, and HalfCheetah from Mujoco), and a real-robot, ball-dispersal task. We published code and videos from our experiments at https://github.com/yuki-kadokawa/cyclic-policy-distillation.
BibTeX
@article{kadokawa2023cyclic,
  author = {Yuki Kadokawa and Lingwei Zhu and Yoshihisa Tsurumine and Takamitsu Matsubara},
  title = {{Cyclic policy distillation: Sample-efficient sim-to-real reinforcement learning with domain randomization}},
  journal = {Robotics Auton. Syst.},
  year = {2023},
  doi = {10.1016/j.robot.2023.104425},
}

Loading related content...

Figure