SkillMimic-V2: Learning Robust and Generalizable Interaction Skills from Sparse and Noisy Demonstrations

Runyi Yu, Yinhuai Wang, Qihan Zhao, Hok Wai Tsui, Jingbo Wang, Ping Tan, Qifeng Chen

Abstract

SkillMimic-V2 is a reinforcement learning (RL) framework designed to enable physically simulated robots to acquire complex interaction skills from sparse and noisy human demonstrations. By addressing the "chain break" problem—where small deviations from a demonstration lead to catastrophic failure—SkillMimic-V2 utilizes a continuous state transition field and adaptive sampling. This allows robots to recover from errors and transition between skills far more robustly than previous methods, bridging the gap between imperfect real-world data and functional robotic control.

Introduction

Reinforcement Learning from Interaction Demonstration (RLID) is a powerful paradigm for teaching robots how to interact with objects by mimicking humans. However, real-world human data is rarely perfect; it is often sparse (capturing only a few specific ways to perform a task) and noisy (containing movements that are physically impossible for a robot to replicate exactly).

Existing methods often struggle when a robot hits a "chain break" point. A small deviation from the demonstration path leads to failure because the robot hasn't learned how to recover from states outside the narrow training data. SkillMimic-V2 introduces a suite of techniques to transform these "broken" trajectories into robust, generalizable skills.

Methodology

The authors introduce four key components to build robustness from imperfect data:

  1. Stitched Trajectory Graph (STG): This technique discovers potential transitions between disconnected skill fragments. For example, it identifies states where a robot could realistically switch from dribbling to shooting, even if that specific transition was never explicitly recorded.
  2. State Transition Field (STF): Instead of learning a single linear path, STF creates a "field" of valid movements. By sampling states in a small "bubble" around the original data and using masked states to bridge gaps, the robot learns to navigate back to a successful trajectory from almost any nearby error state.
  3. Adaptive Trajectory Sampling (ATS): A dynamic curriculum strategy that tracks which parts of a skill the robot struggles with and automatically increases the frequency of those "hard samples" during training.
  4. History Encoder (HE): To handle time-dependent interactions (like timing a basketball release), the HE uses pre-trained temporal embeddings to give the robot a "memory" of past states.
SkillMimic-V2 enables robots to perform diverse tasks, from household object manipulation to fine-grained hand-object interactions.

Experimental Evaluation

The framework was evaluated on two primary benchmarks:

  • BallPlay-M: A basketball dataset featuring high-dynamic skills like multi-directional dribbling, layups, and shooting.
  • ParaHome: A household interaction dataset involving tasks like pouring tea, placing pans, and retrieving books.

Key Results and Insights

SkillMimic-V2 significantly outperformed state-of-the-art baselines like DeepMimic and the original SkillMimic across several metrics:

  • Success Rate (SR): On basketball tasks, the average success rate jumped by 45% compared to previous methods.
  • Skill Transitions (TSR): The ability to switch between skills (e.g., from dribbling to shooting) saw a massive improvement, rising from roughly 15% to over 93%.
  • Generalization: In ParaHome tasks, the model achieved a 35% improvement in handling varied object poses and rotations.
Visualization of complex basketball skill sequences and transitions learned by the agent using the BallPlay-M dataset.

The core insight of the paper is that robustness is functionally the same as generalization. By forcing the RL agent to "inpaint" missing physics between sparse data points, the robot learns the underlying laws of the environment rather than just memorizing a path.

Conclusion and Future Work

SkillMimic-V2 demonstrates that robots can learn robust interaction skills even from highly imperfect data. While effective, the system still faces challenges with heavily corrupted demonstrations where the original intent is lost. Future work may involve integrating large-scale interaction priors from foundation models to provide better guidance in extremely data-sparse environments, potentially moving these skills from simulation to complex, multi-stage real-world tasks.

Summary updated

BibTeX
@inproceedings{yu2025skillmimic,
  author = {Runyi Yu and Yinhuai Wang and Qihan Zhao and Hok Wai Tsui and Jingbo Wang and Ping Tan and Qifeng Chen},
  title = {{SkillMimic-V2: Learning Robust and Generalizable Interaction Skills from Sparse and Noisy Demonstrations}},
  booktitle = {SIGGRAPH (Conference Paper Track)},
  year = {2025},
  doi = {10.1145/3721238.3730640},
}
Figure