Cyclic Policy Distillation: Sample-Efficient Sim-to-Real Reinforcement Learning with Domain Randomization

Yuki Kadokawa, Lingwei Zhu, Yoshihisa Tsurumine, Takamitsu Matsubara

Abstract

Deep reinforcement learning with domain randomization learns a control policy in various simulations with randomized physical and sensor model parameters to become transferable to the real world in a zero-shot setting. However, a huge number of samples are often required to learn an effective policy when the range of randomized parameters is extensive due to the instability of policy updates. To alleviate this problem, we propose a sample-efficient method named cyclic policy distillation (CPD). CPD divides the range of randomized parameters into several small sub-domains and assigns a local policy to each one. Then local policies are learned while cyclically transitioning to sub-domains. CPD accelerates learning through knowledge transfer based on expected performance improvements. Finally, all of the learned local policies are distilled into a global policy for sim-to-real transfers. CPD's effectiveness and sample efficiency are demonstrated through simulations with four tasks (Pendulum from OpenAIGym and Pusher, Swimmer, and HalfCheetah from Mujoco), and a real-robot, ball-dispersal task.

Introduction

To deploy robots in the real world, researchers frequently employ Domain Randomization (DR). By training a policy across a wide spectrum of simulated physical properties—such as friction, mass, or gravity—the resulting agent can ideally handle the "reality gap" during zero-shot deployment. However, standard DR suffers from significant sample inefficiency. When the randomization range is broad, policy updates become unstable as the agent struggles to reconcile highly varied experiences. Existing solutions like curriculum learning or task partitioning often lack efficient knowledge sharing or require extensive manual tuning.

The Cyclic Policy Distillation (CPD) Framework

The proposed CPD method addresses these instabilities by breaking the learning problem into manageable pieces and facilitating knowledge transfer between them.

The CPD framework partitions the randomized domain into sub-domains and uses cyclic learning with knowledge transfer to train local policies before distilling them into a global policy.

The framework consists of three primary stages:

  1. Domain Partitioning: The wide range of randomized parameters is divided into several overlapping sub-domains (e.g., low, medium, and high friction zones).
  2. Cyclic Learning with Knowledge Transfer: The agent trains on one sub-domain at a time, moving cyclically (e.g., $1 \to 2 \to \dots \to N \to \dots \to 1$). To prevent forgetting and accelerate learning, CPD uses a Monotonic Policy Improvement scheme. This "mixes" the current policy with knowledge from the previous sub-domain's neighbor, ensuring stable updates by leveraging environmental similarities.
  3. Global Distillation: After local policies reach expertise in their respective sub-domains, they are distilled into a single global policy $\pi^{(g)}$ capable of handling the entire parameter range.

Experimental Evaluation

The researchers evaluated CPD across various benchmarks, including OpenAI Gym's Pendulum, MuJoCo continuous control tasks (Pusher, Swimmer, HalfCheetah), and a complex real-world "ball-dispersal" task.

Comparison of simulated sub-domains with varying physical parameters and the target real-world domain for the ball-dispersal task.

Key Results

  • Efficiency: In the Pendulum task, CPD demonstrated 3x to 6x better sample efficiency than baseline methods like P2PDRL or Divide-and-Conquer (DnC).
  • Performance: In the MuJoCo HalfCheetah task, CPD was the only method to converge on an effective policy within a practical sample limit, whereas standard Domain Randomization (SAC-DR) failed to learn.
  • Sim-to-Real Transfer: In the real-robot ball-dispersal task using a UR3 arm, the CPD-trained policy achieved a 40% higher reward than baselines. It successfully generalized to different ball types (hard juggling balls vs. soft beads) not encountered during simulation.

Discussion and Future Work

CPD's success stems from its "neighborly advice" approach—exploiting the mathematical similarity between adjacent sub-domains to jump-start learning. By using a dynamic mixture rate based on expected performance, the agent avoids the pitfalls of naive policy copying.

However, some challenges remain. Currently, domain partitioning requires manual design, and the sequential nature of cyclic learning poses hurdles for massive parallelization. Future research may explore automated partitioning and theoretical proofs for monotonic improvement across randomized multi-domains.

Code and videos are available at https://github.com/yuki-kadokawa/cyclic-policy-distillation.

Summary updated

Abstract

Deep reinforcement learning with domain randomization learns a control policy in various simulations with randomized physical and sensor model parameters to become transferable to the real world in a zero-shot setting. However, a huge number of samples are often required to learn an effective policy when the range of randomized parameters is extensive due to the instability of policy updates. To alleviate this problem, we propose a sample-efficient method named cyclic policy distillation (CPD). CPD divides the range of randomized parameters into several small sub-domains and assigns a local policy to each one. Then local policies are learned while cyclically transitioning to sub-domains. CPD accelerates learning through knowledge transfer based on expected performance improvements. Finally, all of the learned local policies are distilled into a global policy for sim-to-real transfers. CPD's effectiveness and sample efficiency are demonstrated through simulations with four tasks (Pendulum from OpenAIGym and Pusher, Swimmer, and HalfCheetah from Mujoco), and a real-robot, ball-dispersal task. We published code and videos from our experiments at https://github.com/yuki-kadokawa/cyclic-policy-distillation.
BibTeX
@article{kadokawa2023cyclic,
  author = {Yuki Kadokawa and Lingwei Zhu and Yoshihisa Tsurumine and Takamitsu Matsubara},
  title = {{Cyclic Policy Distillation: Sample-Efficient Sim-to-Real Reinforcement Learning with Domain Randomization}},
  journal = {Robotics Auton. Syst.},
  year = {2023},
  doi = {10.1016/j.robot.2023.104425},
}
Figure