Abstract
Although reinforcement learning has seen tremendous success recently, this kind of trial-and-error learning can be impractical or inefficient in complex environments. The use of demonstrations, on the other hand, enables agents to benefit from expert knowledge rather than having to discover the best action to take through exploration. In this survey, we discuss the advantages of using demonstrations in sequential decision making, various ways to apply demonstrations in learning-based decision making paradigms (for example, reinforcement learning and planning in the learned models), and how to collect the demonstrations in various scenarios. Additionally, we exemplify a practical pipeline for generating and utilizing demonstrations in the recently proposed ManiSkill robot learning benchmark.
Introduction: The Sample Complexity Challenge
While deep Reinforcement Learning (RL) has achieved superhuman performance in domains like Go and Atari, applying it to real-world robotics or autonomous driving remains a significant challenge. The standard "trial-and-error" approach requires millions of interactions, many of which might result in costly failures or damage to the environment.
The core hurdle is sample complexity: the sheer volume of data needed to learn an optimal policy from scratch. In "sparse reward" environments—where the agent only receives a signal upon total success—an agent might wander aimlessly without ever encountering the goal state. Expert demonstrations act as a "cheat sheet," allowing agents to bypass exhaustive exploration by leveraging pre-existing knowledge.
Methodologies for Utilizing Demonstrations
The survey categorizes the use of demonstrations into two primary phases: offline pre-training and online guidance.
Offline Utilization (Pre-training)
- Policy Learning: Methods like Behavior Cloning (BC) map states directly to actions, while Inverse Reinforcement Learning (IRL) attempts to recover the underlying reward function that motivated the expert.
- Skill Discovery: Long-horizon trajectories can be decomposed into reusable "skills" (e.g., "pick," "place") using latent variable models, providing a more structured action space.
- World Models: Demonstrations help agents learn the environment's transition dynamics, $T(s'|s,a)$. This allows the agent to "plan" or simulate outcomes internally before executing actions in the real world.
- Algorithm Distillation: This novel approach involves training a Transformer on the entire learning history of an agent. By processing sequences of observations, actions, and rewards, the model learns the process of improvement itself.
Online Utilization (Guidance)
- Experience Buffers: Expert transitions are mixed into the agent's replay buffer, ensuring it consistently learns from high-quality data alongside its own exploration.
- Regularization: Agents can be penalized for deviating too far from the expert's "occupancy measure," keeping the learning process within safe or productive regions of the state space.
- Curriculum of Start States: By resetting an agent to various intermediate states found in demonstrations, difficult long-horizon tasks are broken down into manageable segments.
Key Insights and Benchmarks
Modern research is moving beyond simple mimicry. Demonstrations are increasingly used for representation learning, helping agents understand object properties (like handles or buttons) before physical interaction. Furthermore, the distinction between RL and Planning is blurring; agents now frequently learn world models from demonstrations and then apply planning algorithms like Model Predictive Control (MPC) within those models.
The paper highlights several critical benchmarks:
- ManiSkill / ManiSkill2: Focuses on generalizable manipulation with millions of demonstration frames.
- D4RL: A standard for offline RL with varied trajectory quality.
- Ego4D: Provides thousands of hours of egocentric human video for cross-embodiment learning.
Conclusion and Future Directions
Hybrid methods—combining offline pre-training with online fine-tuning—consistently represent the state-of-the-art. However, challenges remain in scaling data collection and solving the "correspondence problem" (transferring human movements to robotic joints). Future work points toward autonomous agents that can selectively ask for human help, creating a more efficient, human-in-the-loop data pipeline.
Summary updated