Robust Policy Learning via Offline Skill Diffusion

Woo Kyung Kim, Minjong Yoo, Honguk Woo

Abstract

Skill-based reinforcement learning (RL) approaches have shown considerable promise, especially in solving long-horizon tasks via hierarchical structures. These skills, learned task-agnostically from offline datasets, can accelerate the policy learning process for new tasks. Yet, the application of these skills in different domains remains restricted due to their inherent dependency on the datasets, which poses a challenge when attempting to learn a skill-based policy via RL for a target domain different from the datasets' domains. In this paper, we present a novel offline skill learning framework, DuSkill, which employs a guided Diffusion model to generate versatile skills extended from the limited skills in datasets, thereby enhancing the robustness of policy learning for tasks in different domains. Specifically, we devise a guided diffusion-based skill decoder in conjunction with hierarchical encoding to disentangle the skill embedding space into two distinct representations: one for encapsulating domain-invariant behaviors and the other for delineating the factors that induce domain variations. Our DuSkill framework enhances the diversity of skills learned offline, enabling accelerated learning of high-level policies for different domains. Experiments show that DuSkill outperforms other skill-based imitation learning and RL algorithms for several long-horizon tasks, demonstrating its benefits in few-shot imitation and online RL.

Introduction

Skill-based reinforcement learning (RL) is a powerful paradigm for solving complex, long-horizon tasks by decomposing them into reusable "skills" extracted from offline datasets. However, these learned skills are often brittle, exhibiting a heavy dependency on the specific domain—including physics, constraints, or reward structures—of the training data. This creates a significant "domain gap" that hinders real-world deployment.

For instance, if a robot learns manipulation skills from a dataset where it was programmed to move slowly for safety, it may fail when tasked with a downstream mission requiring high-speed execution. Conventional methods struggle to adapt because they are typically limited to imitating the specific behaviors present in the training set. Since obtaining comprehensive datasets covering every possible domain variation is practically impossible, a method for extrapolating skills beyond the source distribution is required.

Comparison between conventional skill-based learning and the proposed offline skill diffusion approach.

The DuSkill Framework

We introduce the DuSkill framework, designed to bridge the domain gap by generating versatile skills through a guided diffusion process. The framework operates in two primary phases: the offline skill diffusion phase and the downstream policy learning phase.

Hierarchical Domain Encoding

At the core of DuSkill is a Hierarchical Domain Encoder that disentangles a skill into two distinct embedding spaces:

  • Domain-invariant ($z_\rho$): This captures the fundamental "what" of the behavior, such as the spatial trajectory needed to open a drawer, which remains constant regardless of execution speed.
  • Domain-variant ($z_\sigma$): This captures the "how" or the style of the behavior, such as speed or force, conditioned on specific domain parameters ($\omega$).

Guided Diffusion-based Decoder

Unlike standard Variational Autoencoders (VAEs), DuSkill utilizes a Diffusion model to reconstruct actions. By employing classifier-free guidance, the decoder can modulate how much the domain-variant features influence the final action. By adjusting a guidance weight ($\delta$), the model can effectively generate "new" skills—such as higher-speed trajectories—that were not explicitly represented in the training data. This allows the model to act as a diversity engine, filling in the gaps of the original dataset.

Overview of the DuSkill framework, illustrating the offline skill diffusion phase and the downstream policy learning phase.

Experimental Evaluation

The framework was evaluated using a multi-stage version of the Meta-World benchmark. To test robustness, we introduced three types of domain variations: speed domains (varying time constraints), energy domains (varying energy usage), and wind domains (varying external forces). The difficulty was categorized from "Source" (training domain) to "Level 3," representing the highest disparity from the training data.

Key Results and Insights

DuSkill demonstrated superior robustness and sample efficiency across all tested scenarios:

  • Robustness to Domain Shift: In few-shot imitation tasks, DuSkill maintained high performance even at Level 3 disparity, showing only a 7.6% performance drop, whereas competitive baselines like FIST saw a 25.3% drop.
  • Online RL Adaptation: In the Speed domain, DuSkill outperformed the state-of-the-art SPiRL-c by 89.16% in target domains, proving its ability to adapt to new constraints that traditional skill-based models cannot handle.
  • Efficiency: DuSkill achieves robust performance with as few as 5 trajectories for imitation and shows significantly faster convergence in online RL settings compared to baselines.

The success of DuSkill is attributed to the iterative denoising process of diffusion models, which allows for finer control over skill generation, and the explicit disentanglement of task-core behaviors from domain-specific styles. This enables a form of behavioral extrapolation, where the model understands the components of a skill well enough to adapt them to novel environments.

Conclusion

DuSkill provides a robust framework for skill-based RL by leveraging guided diffusion to overcome the limitations of offline datasets. By separating domain-invariant intent from domain-variant execution, it allows robots to adapt to new physical constraints and requirements with minimal fine-tuning. Future work will explore the framework's performance across extreme embodiment shifts and transfer between entirely different simulation environments.

Summary updated

Abstract

Skill-based reinforcement learning (RL) approaches have shown considerable promise, especially in solving long-horizon tasks via hierarchical structures. These skills, learned task-agnostically from offline datasets, can accelerate the policy learning process for new tasks. Yet, the application of these skills in different domains remains restricted due to their inherent dependency on the datasets, which poses a challenge when attempting to learn a skill-based policy via RL for a target domain different from the datasets' domains. In this paper, we present a novel offline skill learning framework DuSkill which employs a guided Diffusion model to generate versatile skills extended from the limited skills in datasets, thereby enhancing the robustness of policy learning for tasks in different domains. Specifically, we devise a guided diffusion-based skill decoder in conjunction with the hierarchical encoding to disentangle the skill embedding space into two distinct representations, one for encapsulating domain-invariant behaviors and the other for delineating the factors that induce domain variations in the behaviors. Our DuSkill framework enhances the diversity of skills learned offline, thus enabling to accelerate the learning procedure of high-level policies for different domains. Through experiments, we show that DuSkill outperforms other skill-based imitation learning and RL algorithms for several long-horizon tasks, demonstrating its benefits in few-shot imitation and online RL.
BibTeX
@inproceedings{kim2024robust,
  author = {Woo Kyung Kim and Minjong Yoo and Honguk Woo},
  title = {{Robust Policy Learning via Offline Skill Diffusion}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  url = {https://openreview.net/forum?id=FTu42GczsH},
}
Figure