Learning Policies for Embodied Virtual Agents through Demonstration

Jonathan Dinerstein, Parris K. Egbert, Dan VenturaIJCAI 2007

Introduction

Creating lifelike "Embodied Virtual Agents" (EVAs)—software agents with graphical bodies—is a significant challenge in game development and simulation. Traditional methods often rely on manual programming, which is rigid and expertise-intensive, or Reinforcement Learning (RL), which struggles with high-dimensional continuous spaces and the difficulty of defining mathematical reward functions for aesthetic qualities like "graceful movement."

This paper, presented at IJCAI 2007, proposes a "Learning from Demonstration" (LfD) framework. This approach allows non-technical users, such as animators, to "teach" agents by manually controlling them through a simulation. By cleaning human-generated data and generalizing it through machine learning, the system creates reactive policies for complex environments in minutes rather than weeks.

The LfD Framework

The proposed workflow bypasses the need for complex reward structures by mimicking human behavior directly. The process consists of three distinct phases:

The LfD workflow: (a) human demonstration loop and (b) data processing pipeline.
  1. Demonstration: A human user "drives" the agent through the simulation. The system records state-action pairs $\langle s, a \rangle$ at fixed intervals, where $s$ represents environmental variables and $a$ represents control inputs.
  2. Conflict Elimination: To address human inconsistency, the authors introduced a vector-median method. This algorithm identifies and prunes "conflicts"—instances where a demonstrator reacted differently to similar situations due to fatigue or distraction—ensuring the final policy is smooth and decisive.
  3. Generalization: The cleaned data is processed using $\epsilon$-regression Support Vector Machines (SVM) or k-Nearest Neighbor (k-NN) to create a continuous policy mapping ($\mu$), allowing the agent to handle novel situations it did not specifically encounter during training.

Experimental Environments

The system was evaluated using two custom-built test beds designed to challenge the agent's ability to navigate high-dimensional spaces and exhibit natural behavior.

The spaceship pilot test bed, navigating a 9D continuous state space asteroid field.
  • Spaceship Pilot: A 9-dimensional continuous state space where the agent must navigate a dense asteroid field.
  • Virtual Human Crowd: A scenario requiring groups of characters to move naturally and react to dynamic environmental hazards, such as a falling boulder.

Key Results and Insights

The demonstration-based approach showed significant advantages over traditional Reinforcement Learning and search algorithms like A* and PEGASUS:

  • Efficiency: In the spaceship task, the LfD approach required only 8 minutes of human demonstration. In contrast, traditional RL was estimated to take a week of computation and struggled to produce effective policies in the high-dimensional space.
  • Performance: Learned policies, particularly those using SVM, were extremely fast, making decisions in 1–2 microseconds—ideal for real-time applications.
  • Accessibility: A user study confirmed that non-technical creators could learn the system in under four minutes and produce high-quality behaviors within 15 minutes.

The core insight is that aesthetic quality is "baked into" the human demonstration. By prioritizing data cleaning over mathematical optimality, the system achieves "sufficiently good" and visually appealing results far more quickly than "black box" optimization methods.

Conclusion and Future Work

While highly effective, the system has limitations. Agents may struggle in "state space gaps"—areas of the environment never visited by the demonstrator. Furthermore, the current deterministic policies are best suited for reactive tasks rather than multi-stage, goal-directed behaviors. Future work may involve "forcing" agents into unvisited states during training to improve robustness. Overall, the paper demonstrates that "human-in-the-loop" AI offers a powerful, accessible path for creating sophisticated virtual behaviors in creative industries.

Summary updated

BibTeX
@inproceedings{dinerstein2007learning,
  author = {Jonathan Dinerstein and Parris K. Egbert and Dan Ventura},
  title = {{Learning Policies for Embodied Virtual Agents through Demonstration}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2007},
}
Figure