Abstract
Skill-based reinforcement learning (RL) approaches have shown considerable promise, especially in solving long-horizon tasks via hierarchical structures. These skills, learned task-agnostically from offline datasets, can accelerate the policy learning process for new tasks. Yet, the application of these skills in different domains remains restricted due to their inherent dependency on the datasets, which poses a challenge when attempting to learn a skill-based policy via RL for a target domain different from the datasets' domains. In this paper, we present a novel offline skill learning framework, DuSkill, which employs a guided Diffusion model to generate versatile skills extended from the limited skills in datasets, thereby enhancing the robustness of policy learning for tasks in different domains. Specifically, we devise a guided diffusion-based skill decoder in conjunction with hierarchical encoding to disentangle the skill embedding space into two distinct representations: one for encapsulating domain-invariant behaviors and the other for delineating the factors that induce domain variations. Our DuSkill framework enhances the diversity of skills learned offline, enabling accelerated learning of high-level policies for different domains. Experiments show that DuSkill outperforms other skill-based imitation learning and RL algorithms for several long-horizon tasks, demonstrating its benefits in few-shot imitation and online RL.
Introduction
Skill-based reinforcement learning (RL) is a powerful paradigm for solving complex, long-horizon tasks by decomposing them into reusable "skills" extracted from offline datasets. However, these learned skills are often brittle, exhibiting a heavy dependency on the specific domain—including physics, constraints, or reward structures—of the training data. This creates a significant "domain gap" that hinders real-world deployment.
For instance, if a robot learns manipulation skills from a dataset where it was programmed to move slowly for safety, it may fail when tasked with a downstream mission requiring high-speed execution. Conventional methods struggle to adapt because they are typically limited to imitating the specific behaviors present in the training set. Since obtaining comprehensive datasets covering every possible domain variation is practically impossible, a method for extrapolating skills beyond the source distribution is required.
The DuSkill Framework
We introduce the DuSkill framework, designed to bridge the domain gap by generating versatile skills through a guided diffusion process. The framework operates in two primary phases: the offline skill diffusion phase and the downstream policy learning phase.
Hierarchical Domain Encoding
At the core of DuSkill is a Hierarchical Domain Encoder that disentangles a skill into two distinct embedding spaces:
- Domain-invariant ($z_\rho$): This captures the fundamental "what" of the behavior, such as the spatial trajectory needed to open a drawer, which remains constant regardless of execution speed.
- Domain-variant ($z_\sigma$): This captures the "how" or the style of the behavior, such as speed or force, conditioned on specific domain parameters ($\omega$).
Guided Diffusion-based Decoder
Unlike standard Variational Autoencoders (VAEs), DuSkill utilizes a Diffusion model to reconstruct actions. By employing classifier-free guidance, the decoder can modulate how much the domain-variant features influence the final action. By adjusting a guidance weight ($\delta$), the model can effectively generate "new" skills—such as higher-speed trajectories—that were not explicitly represented in the training data. This allows the model to act as a diversity engine, filling in the gaps of the original dataset.
Experimental Evaluation
The framework was evaluated using a multi-stage version of the Meta-World benchmark. To test robustness, we introduced three types of domain variations: speed domains (varying time constraints), energy domains (varying energy usage), and wind domains (varying external forces). The difficulty was categorized from "Source" (training domain) to "Level 3," representing the highest disparity from the training data.
Key Results and Insights
DuSkill demonstrated superior robustness and sample efficiency across all tested scenarios:
- Robustness to Domain Shift: In few-shot imitation tasks, DuSkill maintained high performance even at Level 3 disparity, showing only a 7.6% performance drop, whereas competitive baselines like FIST saw a 25.3% drop.
- Online RL Adaptation: In the Speed domain, DuSkill outperformed the state-of-the-art SPiRL-c by 89.16% in target domains, proving its ability to adapt to new constraints that traditional skill-based models cannot handle.
- Efficiency: DuSkill achieves robust performance with as few as 5 trajectories for imitation and shows significantly faster convergence in online RL settings compared to baselines.
The success of DuSkill is attributed to the iterative denoising process of diffusion models, which allows for finer control over skill generation, and the explicit disentanglement of task-core behaviors from domain-specific styles. This enables a form of behavioral extrapolation, where the model understands the components of a skill well enough to adapt them to novel environments.
Conclusion
DuSkill provides a robust framework for skill-based RL by leveraging guided diffusion to overcome the limitations of offline datasets. By separating domain-invariant intent from domain-variant execution, it allows robots to adapt to new physical constraints and requirements with minimal fine-tuning. Future work will explore the framework's performance across extreme embodiment shifts and transfer between entirely different simulation environments.
Summary updated