Attention Guided Imitation Learning and Reinforcement Learning.

Ruohan ZhangAAAI 2019 · OpenReview

Abstract

We propose a framework that uses learned human visual attention model to guide the learning process of an imitation learning or reinforcement learning agent. We have collected high-quality human action and eye-tracking data while playing Atari games in a carefully controlled experimental setting. We have shown that incorporating a learned human gaze model into deep imitation learning yields promising results.

Introduction: The Challenge of Sample Inefficiency

End-to-end learning for visuomotor behaviors, such as playing video games or navigating complex environments, typically requires millions of training samples to reach expert-level performance. This "sample inefficiency" remains a significant bottleneck in artificial intelligence.

Humans, by contrast, are remarkably efficient learners. This efficiency stems from a selective attention mechanism; we do not process every pixel in our visual field equally. Instead, our eyes focus on task-relevant features, effectively filtering out high-dimensional noise. This research addresses how to capture human gaze behavior and utilize it as a powerful feature extractor to enhance the learning speed and performance of artificial agents.

Methods: Integrating Human Gaze

The proposed framework explores several interconnected approaches to incorporate human visual attention into the learning process:

Attention Guided Imitation Learning (AGIL)

AGIL is a two-stage framework. First, a gaze network is trained on human eye-tracking data to predict where a human expert would look given a specific game state. Second, a policy network is trained to predict actions. This policy network uses the output of the gaze network to highlight relevant visual features, initially through element-wise masking of the input image.

Visualization of the attention map used in Deep RL with attention to highlight task-relevant features.

Modular RL and Inverse Reinforcement Learning (IRL)

For complex tasks, a modular approach is employed where tasks are decomposed into simpler sub-tasks with factorized state spaces. A Modular Inverse Reinforcement Learning (IRL) algorithm was developed to infer the "intrinsic rewards" humans associate with different objects and goals, based on observed behavior in virtual reality environments.

Fusion Architectures

The research also investigates optimal "fusion" architectures for integrating attention maps into policy networks. Beyond simple masking, the study explores different network layers and arithmetic operators to more effectively combine gaze information with visual input.

Key Insights: Gaze as a Strategic Prior

  • Informative Priors: Human gaze is not random; it encodes object priority and long-term reward strategies. It provides a crucial "hint" to the agent regarding which parts of a high-dimensional state space are most relevant.
  • Bridging the Representation Gap: Standard Deep RL models (like DQN) often struggle with multi-task games (e.g., Seaquest) when learning directly from raw pixels. Attention-guided models bridge the gap between raw pixel data and the modular, feature-based representations humans naturally employ.
  • Biological Inspiration: Informed by neuroscience, this work explores whether "attention gain effects"—the allocation of additional neural resources to specific tasks—can be replicated in artificial neural networks.

Key Results

The AGIL framework demonstrated substantial performance gains over standard imitation learning (IL) across various Atari titles.

Performance comparison (mean scores) between standard imitation learning and the proposed AGIL framework across multiple Atari games.
  • Seaquest: AGIL achieved a score of 788.9, significantly higher than the 252.2 achieved by plain IL on the same dataset and the 195.6 reported in previous benchmarks.
  • MsPacman: AGIL reached a score of 1755.1, outperforming the plain IL score of 1069.9.
  • General Performance: In nearly every game tested (including Breakout, Freeway, and Enduro), the attention-guided approach yielded higher mean scores than standard imitation learning.
  • Modular IRL: The proposed modular IRL significantly outperformed standard Bayesian IRL in modeling human-like walking trajectories in novel environments.

Limitations and Future Directions

While promising, several challenges remain:

  • Refining Fusion: The current "naive pipeline" of pixel masking is likely suboptimal. Identifying the most effective architecture to fuse gaze and policy data is an ongoing area of research.
  • Expansion to Reinforcement Learning: While current results primarily focus on Imitation Learning, future work aims to fully integrate these gaze models into Deep Reinforcement Learning frameworks.
  • Self-Sourced Attention: A long-term goal is to move beyond the requirement for human gaze data. The objective is to develop agents capable of learning their own attention mechanisms from experience, mimicking the biological process of learning what to prioritize and what to ignore.

Summary updated

Abstract

We propose a framework that uses learned human visual attention model to guide the learning process of an imitation learning or reinforcement learning agent. We have collected high-quality human action and eye-tracking data while playing Atari games in a carefully controlled experimental setting. We have shown that incorporating a learned human gaze model into deep imitation learning yields promising results.
BibTeX
@inproceedings{zhang2019attention,
  author = {Ruohan Zhang},
  title = {{Attention Guided Imitation Learning and Reinforcement Learning.}},
  booktitle = {AAAI},
  year = {2019},
  url = {https://openreview.net/forum?id=rQYNRfzgdpH},
}
Figure