In-Context Policy Adaptation via Cross-Domain Skill Diffusion

Minjong Yoo, Woo Kyung Kim, Honguk Woo

Introduction

Reinforcement Learning (RL) agents often exhibit extreme brittleness when faced with domain shifts. A policy trained to operate a robotic arm in a specific simulation environment frequently fails when transferred to a setting with different physical parameters—such as gravity or friction—or a different robotic embodiment. Traditionally, addressing these disparities requires gradient-based fine-tuning on target-domain data. However, this approach is often impractical due to extreme data scarcity in new environments and the high computational cost of real-time model updates, which can also lead to catastrophic forgetting.

To overcome these hurdles, we present ICPAD (In-Context Policy Adaptation via Cross-Domain Skill Diffusion). ICPAD is a framework designed for in-context adaptation, allowing agents to adjust their behavior to novel domains purely through input prompts, without any parameter updates. By leveraging diffusion-based skill learning and a middle-tier of domain-agnostic prototype skills, ICPAD achieves state-of-the-art performance across complex, long-horizon tasks.

Methodology

ICPAD operates through a two-phase architecture that decouples high-level intent from low-level execution.

1. Offline Learning: Establishing Prototype Skills

During the pre-training phase, the agent learns a library of Prototype Skills that serve as a universal behavioral language. This is achieved through two primary components:

  • Domain-Agnostic Skill Encoder ($\Phi_E$): This module extracts high-level behavior patterns (e.g., "grasping") that remain consistent across disparate domains.
  • Domain-Grounded Skill Adapter ($\Phi_A$): A diffusion-based decoder translates these abstract prototypes into specific actions tailored to a given domain. The use of Diffusion Models is critical here; unlike standard Gaussian models, diffusion can accurately represent the complex, multi-modal distributions inherent in expert behavior.

To ensure the skills remain truly domain-agnostic, the framework employs specific consistency losses ($L_{cross-E}$ and $L_{cross-A}$). These losses force the latent skill representations to remain invariant, regardless of the source domain's specific physics or noise profiles.

2. In-Context Adaptation: Dynamic Domain Prompting

When deployed in a new target domain, the agent utilizes Dynamic Domain Prompting to align its behavior without retraining:

  • Retriever: The system analyzes a small set of few-shot demonstrations from the target domain.
  • Attention Mechanism: A domain encoder compares the agent's current state to these demonstrations, retrieving the most relevant prompts to guide the skill adapter. This allows the agent to "understand" the target domain's constraints and adjust its action outputs on the fly.

Experimental Evaluation

We evaluated ICPAD on two demanding benchmarks designed to test robustness under domain disparity:

  1. Metaworld & Multi-stage Metaworld: These robotics manipulation tasks (e.g., drawer closing, button pressing) were modified with varying levels of action noise and simulated wind to create significant environmental shifts.
  2. CARLA: A high-fidelity autonomous driving simulator used to test adaptation across different vehicle embodiments (car models) and varying weather conditions.

Results and Key Insights

Superior Performance and Robustness

ICPAD demonstrated significant performance gains over existing state-of-the-art methods. In Metaworld tasks with high environmental disparity, ICPAD achieved an 81.6% success rate, whereas the leading baseline, DCMRL, dropped to 51.8%. In the CARLA driving benchmarks, ICPAD outperformed baselines by 11.6% to 21.6% in normalized return when adapting to new vehicles and weather.

Data Efficiency and Horizon Stability

The framework is remarkably data-efficient. Even when demonstrations were available for only 8% of the tasks in the target domain, ICPAD maintained an 83.8% success rate. In contrast, competing methods like DCMRL saw performance plummet to 49.9%. Furthermore, ICPAD proved robust in multi-stage, long-horizon tasks, maintaining a 96.6% success rate where other methods struggled to maintain consistency over time.

The Role of Diffusion

A key takeaway from our research is the efficacy of Diffusion Models as "translators." While prior work often relied on simple Gaussian distributions for skill modeling, ICPAD shows that diffusion is significantly more robust for translating high-level intent into low-level actions, particularly in complex sequences where precision is paramount.

Conclusion and Future Work

ICPAD introduces a paradigm shift in policy adaptation by treating prototype skills as a "lingua franca" across domains. This allows for "domain-wise" adaptation: once the agent understands the underlying physics of a new environment, it can apply that knowledge across its entire repertoire of tasks without further fine-tuning.

Future research will focus on expanding the framework to handle multi-modal data (such as vision and language) and improving the semantic interpretability of the latent prototype skills. Additionally, we aim to validate the "zero-update" adaptation capabilities on physical robotic hardware to test the system's resilience against real-world sensor noise.

Summary updated

BibTeX
@inproceedings{yoo2025context,
  author = {Minjong Yoo and Woo Kyung Kim and Honguk Woo},
  title = {{In-Context Policy Adaptation via Cross-Domain Skill Diffusion}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i21.34373},
}
Figure