Introduction
In Deep Reinforcement Learning (DRL), "skill discovery" allows agents to explore environments and learn useful behaviors—such as running or jumping—without explicit rewards. Ideally, these "primitive skills" should be reusable for complex new tasks, a process known as skill transfer. However, current discovery algorithms, such as Diversity Is All You Need (DIAYN), often produce highly correlated skills. For example, a robot tasked with learning six distinct ways to move might instead learn six minor variations of leaning forward. This redundancy creates a high-dimensional, bloated search space that significantly hinders transfer efficiency in target environments.
The Redundancy Problem
To quantify this issue, researchers applied Principal Component Analysis (PCA) to sets of primitive skills discovered in the MuJoCo physics engine. As shown in Figure 1, the results confirmed massive statistical redundancy: for a set of 6 primitive skills, just 3 eigenvalues accounted for over 98% of the variance. This suggests that the "alphabet" of motion provided to the agent is unnecessarily repetitive, forcing the transfer policy to navigate a redundant and inefficient action space.
Methodology: Independent Skill Transfer (IST)
To address this, the authors proposed a two-stage framework: Learning Independent Skills (LIS) and Independent Skill Transfer (IST). The core idea is to "unmix" correlated behaviors into a compact set of statistically independent "atomic" skills.
1. Learning Independent Skills (LIS)
First, a standard discovery algorithm (DIAYN) identifies primitive skills in a source environment (e.g., HalfCheetah-v3). The actions generated by these skills are then processed using FastICA (Independent Component Analysis) to extract independent components. Finally, a new set of policy networks is trained via supervised learning to represent these refined, independent atomic skills.
2. Independent Skill Transfer (IST)
In the target environment, the agent learns a transfer policy that outputs weights ($\alpha$) and a bias ($b$) to form a linear combination of the independent skills. Because the number of independent skills is significantly lower than the original set, the search space for the transfer policy is reduced, making learning faster and more robust.
Experimental Evaluation
The IST method was tested against direct Primitive Skill Transfer (PST), Primitive Skill Selection (PSS), and learning from scratch (SAC) across three custom MuJoCo benchmarks: jumping hurdles (HCH), climbing slopes (HCA), and navigating stairs (HCU).
- Success Rate: In the Hurdle task, IST achieved a 93.2% success rate, significantly outperforming the next best transfer method (PST) at 73.5%.
- Efficiency: In the Upstair task, IST converged to a high reward in fewer than 150 episodes, whereas PST failed to converge entirely within the same timeframe.
- Robustness: As task difficulty increased (e.g., steeper slopes), IST demonstrated the least performance degradation, suggesting superior generalization to out-of-distribution challenges.
Conclusion and Future Work
Independent Skill Transfer demonstrates that reducing redundancy is critical for efficient reinforcement learning. By distilling primitive behaviors into independent atomic components, agents can solve complex tasks with far fewer samples. While the current framework assumes linear combinations of skills—which proved effective for the HalfCheetah—future work may explore non-linear decomposition methods to handle higher-dimensional robots, such as humanoids, or complex manipulation tasks.
Summary updated