Representations for Action Selection Learning from Real-Time Observation of Task Experts

Mark A. Wood, Joanna BrysonIJCAI 2007

Introduction

Program-level imitation—the process of learning the high-level structure of a task by observing an expert—presents a significant challenge in artificial intelligence due to its inherent combinatorial complexity. This paper describes an evolution of the COntinuous Imitation Learning (COIL) system. While previous iterations of COIL successfully navigated simple environments, they faced significant scaling bottlenecks. The authors address these by replacing heuristic filters with a Bayesian Multi-Layer Perceptron (MLP) framework, transforming action selection into a rigorous probabilistic classification problem.

The COIL Pipeline and its Bottlenecks

The original COIL system followed a multi-stage pipeline designed to extract meaningful events from raw sensor data.

The COIL processing pipeline, showing the progression from raw sensor data to meaningful event items (MEitems).

As shown in Figure 1, the system segments raw data into channels and events, eventually filtering them to find "MEitems" (Meaningful Event items). However, the original "back-end" filters—specifically Recurrence and Mutual Information filters—suffered from cubic algorithmic complexity and required storing an ever-expanding history of events. Furthermore, the system's reliance on histograms for continuous data struggled with the discrete, categorical nature of robotic actions, and its frequentist approximations were highly sensitive to noise.

Bayesian MLP for Action Selection

To create a more scalable and robust system, the authors replaced the final stages of the pipeline with a Bayesian MLP. This architectural shift moves the system from associative learning (storing every perception-action pair) to functional learning (learning a mapping function).

Key Methodological Components

  • Input Encoding: Categorical perceptual data is represented using "1-of-c" (one-hot) bit-vectors.
  • Architecture: The network utilizes a fully connected feedforward structure with three hidden units and a softmax activation function. This ensures that the output represents the posterior probability of an action class.
  • Bayesian Regularization: The model employs MacKay’s "evidence approximation" to re-estimate hyperparameters, providing a principled way to manage model complexity.
  • Automatic Relevance Determination (ARD): By assigning a hyperprior to each input channel, the system can automatically identify and ignore irrelevant sensory data. If an input's weight distribution shows low variance (high coefficient), it is deemed irrelevant.
  • Loss Matrices: The framework allows for the incorporation of prior knowledge via a loss matrix, enabling the system to penalize specific errors (e.g., friendly fire) more heavily than others.

Experimental Evaluation

The researchers utilized the 3D first-person shooter Unreal Tournament (UT) as a complex, real-time sandbox. Two primary tasks were evaluated:

  1. Task 1 (Health Vials): An agent observes an expert collecting items using various movement tactics (Clockwise, Anti-clockwise, and Mixed).
  2. Task 2 (Search and Destroy): A complex combat scenario requiring the agent to distinguish between enemies and friends.

Results and Insights

The performance of the Bayesian MLP was benchmarked against the original COIL algorithms and the C4.5 decision tree algorithm.

Experimental results showing (a) Task 1 performance, (b) Task 2 performance with different loss configurations, and (c) ARD validation identifying relevant sensors.

As illustrated in Figure 3, the MLP achieved near-perfect behavior in Task 1, matching the performance of C4.5. In Task 2, the introduction of a Loss Matrix significantly improved performance by penalizing premature firing, demonstrating how decision theory can modulate agent behavior. Finally, the ARD validation (Figure 3c) confirmed that the system could successfully distinguish between relevant inputs (relative bearing to target) and irrelevant noise (absolute compass bearing).

Conclusion

By adopting a Bayesian MLP framework, the authors successfully addressed the scaling and noise-sensitivity issues of the original COIL system. This approach provides a mathematically rigorous method for selective attention through ARD and allows for the integration of expert prior knowledge via loss matrices. Future work points toward hierarchical systems where multiple MLPs manage local task domains, further enhancing the agent's ability to handle dynamic, multi-faceted environments.

Summary updated

BibTeX
@inproceedings{wood2007representations,
  author = {Mark A. Wood and Joanna Bryson},
  title = {{Representations for Action Selection Learning from Real-Time Observation of Task Experts}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2007},
}
Figure