Abstract
Skill-based reinforcement learning (RL) approaches have shown considerable promise, especially in solving long-horizon tasks via hierarchical structures. These skills, learned task-agnostically from offline datasets, can accelerate the policy learning process for new tasks. Yet, the application of these skills in different domains remains restricted due to their inherent dependency on the datasets, which poses a challenge when attempting to learn a skill-based policy via RL for a target domain different from the datasets' domains. In this paper, we present a novel offline skill learning framework DuSkill which employs a guided Diffusion model to generate versatile skills extended from the limited skills in datasets, thereby enhancing the robustness of policy learning for tasks in different domains. Specifically, we devise a guided diffusion-based skill decoder in conjunction with the hierarchical encoding to disentangle the skill embedding space into two distinct representations, one for encapsulating domain-invariant behaviors and the other for delineating the factors that induce domain variations in the behaviors. Our DuSkill framework enhances the diversity of skills learned offline, thus enabling to accelerate the learning procedure of high-level policies for different domains. Through experiments, we show that DuSkill outperforms other skill-based imitation learning and RL algorithms for several long-horizon tasks, demonstrating its benefits in few-shot imitation and online RL.
Introduction
Skill-based Reinforcement Learning (RL) is a powerful paradigm for solving complex, long-horizon tasks by decomposing them into reusable "skills" such as "open drawer" or "pick up object." However, the utility of these skills is often constrained by the offline datasets from which they are derived. These datasets are typically tied to specific "domains" — environmental conditions like robot speed or external forces.
When a robot trained in a "safety-first" (slow) domain is tasked with performing in a "stringent time" (fast) domain, existing methods often fail. This "domain gap" occurs because learned skills are limited to the specific behaviors present in the training data, lacking the diversity required for adaptation. DuSkill addresses this by using diffusion models to "hallucinate" and extend the skill space beyond the original dataset.
Methodology: The DuSkill Framework
DuSkill combines hierarchical representation learning with the generative capabilities of diffusion models to bridge the domain gap.
Hierarchical Domain Encoding
To handle domain shifts, DuSkill disentangles a skill into two distinct latent representations:
- Domain-Invariant ($z_\rho$): Captures the fundamental "what" of the behavior (e.g., the trajectory required to slide a puck).
- Domain-Variant ($z_\sigma$): Captures the "how" based on environment factors (e.g., adjusting force for wind or speed for time limits).
Guided Diffusion-based Decoder
Unlike standard Variational Autoencoders (VAEs) that may suffer from posterior collapse and generate "average" behaviors, DuSkill employs a Denoising Diffusion Probabilistic Model (DDPM). By utilizing classifier-free guidance, the decoder blends invariant and variant embeddings to generate compatible skills for domains it hasn't seen before.
Learning Phases
The framework operates in two stages:
- Offline Pre-training: Encoders and the diffusion decoder are trained on multi-domain datasets to learn the underlying structure of skills.
- Downstream Adaptation: A high-level policy is trained via few-shot imitation or online RL to select the optimal combination of $z_\rho$ and $z_\sigma$ for a specific target task.
Experimental Evaluation
The researchers evaluated DuSkill on a custom Multi-stage Meta-World benchmark involving sequential sub-tasks (e.g., "slide puck -> close drawer -> push button -> open door"). Environmental shifts included variations in speed, energy (force constraints), and wind (kinematic disturbances).
Key Results
- Robust Adaptation: In few-shot imitation (using as few as 3 trajectories), DuSkill showed only a 7.6% performance degradation across domains, compared to 25.3% for competitive baselines like FIST.
- Online RL Performance: DuSkill outperformed the state-of-the-art SPiRL-c by 89.16% when fine-tuning in a new "Speed" domain.
- Sample Efficiency: High success rates were achieved with only 50k samples, a threshold where many baselines failed to show significant improvement.
Conclusion and Future Work
DuSkill demonstrates that disentangling core behaviors from domain-specific variations through diffusion models significantly enhances the robustness and diversity of robotic skills. While currently validated in simulation for parameter shifts like speed and wind, future work will explore more extreme shifts, such as different robot embodiments and more complex, non-linear task sequences in real-world environments.
Summary updated