Independent Skill Transfer for Deep Reinforcement Learning

Qiangxing Tian, Guanchu Wang, Jinxin Liu, Donglin Wang, Yachen Kang

Introduction

In Deep Reinforcement Learning (DRL), "skill discovery" allows agents to explore environments and learn useful behaviors—such as running or jumping—without explicit rewards. Ideally, these "primitive skills" should be reusable for complex new tasks, a process known as skill transfer. However, current discovery algorithms, such as Diversity Is All You Need (DIAYN), often produce highly correlated skills. For example, a robot tasked with learning six distinct ways to move might instead learn six minor variations of leaning forward. This redundancy creates a high-dimensional, bloated search space that significantly hinders transfer efficiency in target environments.

The Redundancy Problem

To quantify this issue, researchers applied Principal Component Analysis (PCA) to sets of primitive skills discovered in the MuJoCo physics engine. As shown in Figure 1, the results confirmed massive statistical redundancy: for a set of 6 primitive skills, just 3 eigenvalues accounted for over 98% of the variance. This suggests that the "alphabet" of motion provided to the agent is unnecessarily repetitive, forcing the transfer policy to navigate a redundant and inefficient action space.

PCA analysis showing that a small number of eigenvalues account for the vast majority of variance in primitive skills, highlighting significant redundancy.

Methodology: Independent Skill Transfer (IST)

To address this, the authors proposed a two-stage framework: Learning Independent Skills (LIS) and Independent Skill Transfer (IST). The core idea is to "unmix" correlated behaviors into a compact set of statistically independent "atomic" skills.

The two-stage framework for Independent Skill Transfer (IST), moving from skill discovery to independent skill refinement and final transfer.

1. Learning Independent Skills (LIS)

First, a standard discovery algorithm (DIAYN) identifies primitive skills in a source environment (e.g., HalfCheetah-v3). The actions generated by these skills are then processed using FastICA (Independent Component Analysis) to extract independent components. Finally, a new set of policy networks is trained via supervised learning to represent these refined, independent atomic skills.

2. Independent Skill Transfer (IST)

In the target environment, the agent learns a transfer policy that outputs weights ($\alpha$) and a bias ($b$) to form a linear combination of the independent skills. Because the number of independent skills is significantly lower than the original set, the search space for the transfer policy is reduced, making learning faster and more robust.

Experimental Evaluation

The IST method was tested against direct Primitive Skill Transfer (PST), Primitive Skill Selection (PSS), and learning from scratch (SAC) across three custom MuJoCo benchmarks: jumping hurdles (HCH), climbing slopes (HCA), and navigating stairs (HCU).

  • Success Rate: In the Hurdle task, IST achieved a 93.2% success rate, significantly outperforming the next best transfer method (PST) at 73.5%.
  • Efficiency: In the Upstair task, IST converged to a high reward in fewer than 150 episodes, whereas PST failed to converge entirely within the same timeframe.
  • Robustness: As task difficulty increased (e.g., steeper slopes), IST demonstrated the least performance degradation, suggesting superior generalization to out-of-distribution challenges.

Conclusion and Future Work

Independent Skill Transfer demonstrates that reducing redundancy is critical for efficient reinforcement learning. By distilling primitive behaviors into independent atomic components, agents can solve complex tasks with far fewer samples. While the current framework assumes linear combinations of skills—which proved effective for the HalfCheetah—future work may explore non-linear decomposition methods to handle higher-dimensional robots, such as humanoids, or complex manipulation tasks.

Summary updated

Abstract

Recently, diverse primitive skills have been learned by adopting the entropy as intrinsic reward, which further shows that new practical skills can be produced by combining a variety of primitive skills. This is essentially skill transfer, very useful for learning high-level skills but quite challenging due to the low efficiency of transferring primitive skills. In this paper, we propose a novel efficient skill transfer method, where we learn independent skills and only independent components of skills are transferred instead of the whole set of skills. More concretely, independent components of skills are obtained through independent component analysis (ICA), which always have a smaller amount (or lower dimension) compared with their mixtures. With a lower dimension, independent skill transfer (IST) exhibits a higher efficiency on learning a given task. Extensive experiments including three robotic tasks demonstrate the effectiveness and high efficiency of our proposed IST method in comparison to direct primitive-skill transfer and conventional reinforcement learning.
BibTeX
@inproceedings{tian2020independent,
  author = {Qiangxing Tian and Guanchu Wang and Jinxin Liu and Donglin Wang and Yachen Kang},
  title = {{Independent Skill Transfer for Deep Reinforcement Learning}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.24963/ijcai.2020/401},
}
Figure