Towards Interpretable Reinforcement Learning Using Attention Augmented Agents

Alex Mott, Daniel Zoran, Mike Chrzanowski, Daan Wierstra, Danilo J. Rezende

Introduction

Deep Reinforcement Learning (RL) agents have achieved remarkable success in complex domains, yet their internal decision-making processes remain largely opaque. Traditional "black box" models process visual inputs through dense layers of convolutions, making it difficult to diagnose failures or extract strategies. While post-hoc visualization techniques like saliency maps exist, they often provide noisy signals that reflect reactions rather than intent.

Researchers from DeepMind propose a shift toward intrinsic interpretability. By integrating a top-down, soft-attention mechanism that acts as a functional bottleneck, they developed an agent that effectively communicates its focus. This architecture forces the agent to sequentially query its environment, making its internal logic directly visible to humans through clear attention maps without sacrificing state-of-the-art performance.

Methods: The Attention Augmented Agent

The proposed architecture consists of three primary components designed to create a structured information flow:

  1. Vision Core: A combination of a Convolutional Neural Network (CNN) and a ConvLSTM processes raw pixels into "Keys" and "Values," mirroring the structure of Transformer architectures.
  2. Query Network: A high-level LSTM serves as the agent's "brain," generating queries based on its internal state and goals. This top-down approach ensures the agent looks for information relevant to its current strategy.
  3. The Bottleneck (The "Answer"): Queries are compared against Keys to produce an attention map. This map weights the Values, which are then summed into a single "Answer" vector. This spatial summation creates a severe bottleneck, forcing the agent to prioritize specific visual information.

To preserve spatial awareness despite the summation bottleneck, the authors append a fixed Fourier spatial basis to the Keys and Values. This allows the agent to reason about absolute positions (e.g., "bottom-left corner") even after spatial compression.

The Attention Augmented Agent architecture featuring a top-down query network and a spatial attention bottleneck.

Experimental Evaluation

The model was evaluated using the Arcade Learning Environment (ALE) across a suite of 57 Atari 2600 games. This benchmark tests the agent's ability to generalize across diverse visual and strategic challenges.

Results and Discussion

The Attention Augmented Agent achieved a 407.1% median human-normalized score, significantly outperforming both feedforward (284.5%) and standard LSTM (45.0%) baselines. Beyond performance, the architecture yielded several key insights into agent behavior:

Emergent Interpretable Strategies

The attention maps revealed sophisticated, human-like strategies that emerged naturally during training:

  • Trip Wires: In games like Breakout and Space Invaders, the agent learned to place fixed points of attention—"trip wires"—that wait for objects like balls or bullets to cross them before triggering an action.
  • Path Planning: In maze games like Ms. Pacman, the agent was observed "scanning" ahead along potential paths to check for hazards, effectively visualizing its internal planning process.
  • Content vs. Spatial Queries: Analysis of query vectors showed a natural separation between "What is at this location?" (content-based) and "Where is this specific object?" (spatial-based) queries.
Emergent 'trip wire' attention patterns in Breakout (top) and Space Invaders (bottom).

Robustness and Validation

To ensure the agent was truly utilizing the attention mechanism, the researchers conducted "masking" experiments, allowing the agent to see only the top 10% of attended regions. Performance remained nearly identical, confirming that the information within the attention maps is sufficient for decision-making. Furthermore, the agent demonstrated robustness to novelty; when a "fake" object was spliced into Seaquest, the agent correctly identified and reacted to it as a threat via its attention mechanism.

Conclusion

The Attention Augmented Agent demonstrates that interpretability does not have to come at the cost of performance. By enforcing a top-down attention bottleneck, the model provides a window into the agent's "thoughts," revealing emergent strategies like trip wires and forward planning. Future work may explore scaling this architecture to high-resolution 3D environments and refining the spatial coordinate systems for more complex motor control tasks.

Summary updated

BibTeX
@inproceedings{mott2019towards,
  author = {Alex Mott and Daniel Zoran and Mike Chrzanowski and Daan Wierstra and Danilo J. Rezende},
  title = {{Towards Interpretable Reinforcement Learning Using Attention Augmented Agents}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2019},
  url = {https://openreview.net/forum?id=-dgkkwecbW6},
}
Figure