KOI: Accelerating Online Imitation Learning via Hybrid Key-state Guidance

Jingxian Lu, Wenke Xia, Dong Wang, Zhigang Wang, Bin Zhao, Di Hu, Xuelong Li

Introduction

Imitation Learning (IL) has emerged as a powerful paradigm for teaching robots complex tasks by leveraging expert demonstrations. While offline IL is data-intensive, Online Imitation Learning (OIL) offers a more dynamic approach, allowing robots to refine their policies through active exploration and trial-and-error. However, OIL is often hampered by a significant "exploration gap." In the vast state-action space, robots struggle to identify which movements are truly progress toward a goal.

State-of-the-art OIL methods typically utilize Optimal Transport (OT) to calculate rewards based on the distance between a robot's trajectory and an expert's video. A critical limitation of these methods is their lack of "task awareness"; they treat every frame of an expert demonstration with equal importance. This uniform weighting fails to distinguish between critical milestones—like grasping an object—and less significant movements, often leading to sub-optimal policies.

The KOI Framework

To address these challenges, the authors propose KOI (Hybrid Key-state guided Online Imitation). KOI accelerates learning by decomposing tasks into two distinct types of key-states that guide the robot's exploration more effectively.

Overview of the KOI framework showing semantic and motion decomposition modules.

Hybrid Key-state Decomposition

  1. Semantic Key-states ("What to do"): The Semantic Decomposition Module (SDM) leverages Large Multimodal Models (LMMs) like GPT-4v. By processing task descriptions and expert videos, the LMM identifies semantic subgoals (e.g., "Pick up the cube") and pinpoint the exact frames where these milestones occur.
  2. Motion Key-states ("How to do"): Between semantic milestones, the Motion Capture Module (MCM) identifies frames with high physical intensity using the Farneback optical flow algorithm. These frames represent the mechanical transitions necessary to move between semantic states.

Importance Weight Adaption

Rather than assuming a uniform distribution of importance across a trajectory, KOI employs a Gaussian Mixture Model (GMM) to assign higher "importance weights" to these identified key-states. This creates a denser, more informative reward signal, effectively bridging the exploration gap by focusing the robot's attention on the most critical parts of the task.

Experimental Evaluation

KOI was rigorously tested across several simulated and real-world domains, including Meta-World (6 manipulation tasks), LIBERO (3 long-horizon tasks), and real-world experiments with an XARM robot.

Success rate comparison across Meta-World and LIBERO benchmarks, showing KOI's superior sample efficiency.

Key Results

  • Superior Sample Efficiency: In both Meta-World and LIBERO, KOI significantly outperformed the previous state-of-the-art (ROT) in both final success rates and learning speed.
  • Real-World Robustness: In physical robot trials, KOI achieved a 100% success rate on a "take the rag off" task, compared to 70% for the baseline. It also showed marked improvement in high-precision tasks like placing a cube on a scale.
  • Stability: Analysis showed that KOI leads to continuously decreasing variance during the learning process, indicating a more stable and predictable convergence than uniform OT methods.

Discussion and Future Work

Ablation studies confirmed that the hybrid nature of KOI is essential; using only semantic or only motion guidance resulted in inferior performance. This aligns with cognitive neuroscience theories suggesting that human learning relies on understanding both objectives and the mechanisms to achieve them.

Despite its success, KOI faces a "cold start" challenge where the initial policy and critic networks are untrained, potentially leading to inefficient early exploration. Future research may explore integrating offline reinforcement learning to pre-train these components, further accelerating the transition to high-performance online refinement.

Summary updated

Abstract

Online Imitation Learning methods struggle with the gap between extensive online exploration space and limited expert trajectories, which hinder efficient exploration due to inaccurate task-aware reward estimation. Inspired by the findings from cognitive neuroscience that task decomposition could facilitate cognitive processing for efficient learning, we hypothesize that an agent could estimate precise task-aware imitation rewards for efficient online exploration by decomposing the target task into the objectives of "what to do" and the mechanisms of "how to do". In this work, we introduce the hybrid Key-state guided Online Imitation (KOI) learning approach, which leverages the integration of semantic and motion key states as guidance for task-aware reward estimation. Initially, we utilize the visual-language models to segment the expert trajectory into semantic key states, indicating the objectives of "what to do". Within the intervals between semantic key states, optical flow is employed to capture motion key states to understand the process of "how to do". By integrating a thorough grasp of both semantic and motion key states, we refine the trajectory-matching reward computation, encouraging task-aware exploration for efficient online imitation learning. Our experiment results prove that our method is more sample efficient than previous state-of-the-art approaches in the Meta-World and LIBERO environments. We also conduct real-world robotic manipulation experiments to validate the efficacy of our method, demonstrating the practical applicability of our KOI method.
BibTeX
@inproceedings{lu2024koi,
  author = {Jingxian Lu and Wenke Xia and Dong Wang and Zhigang Wang and Bin Zhao and Di Hu and Xuelong Li},
  title = {{KOI: Accelerating Online Imitation Learning via Hybrid Key-state Guidance}},
  booktitle = {Conference on Robot Learning},
  year = {2024},
  url = {https://openreview.net/forum?id=JZzaRY8m8r},
}
Figure