Abstract: Visual saliency has emerged as a major visualization tool for interpreting deep reinforcement learning (RL) agents. However, much of the existing research uses it as an analyzing tool rather than an inductive bias for policy learning. In this work, we use visual attention as an inductive bias for RL agents. We propose a novel self-supervised attention learning approach which can 1. learn to select regions of interest without explicit annotations, and 2. act as a plug for existing deep RL methods to improve the learning performance. We empirically show that the self-supervised attention-aware deep RL methods outperform the baselines in the context of both the rate of convergence and performance. Furthermore, the proposed self-supervised attention is not tied with specific policies, nor restricted to a specific scene. We posit that the proposed approach is a general self-supervised attention module for multi-task learning and transfer learning, and empirically validate the generalization ability of the proposed method. Finally, we show that our method learns meaningful object keypoints highlighting improvements both qualitatively and quantitatively.
Introduction
Standard Deep Reinforcement Learning (RL) agents typically process visual observations through a Convolutional Neural Network (CNN) that treats every pixel with equal importance. This contrasts sharply with human perception, which selectively focuses on salient objects—such as characters, enemies, or goals—while filtering out static background noise.
Existing attempts to integrate attention into RL generally rely on top-down mechanisms driven by reward signals, which often fail to generalize, or supervised saliency methods that require expensive human-annotated datasets. This paper introduces a bottom-up, self-supervised approach that allows agents to learn what constitutes a "region of interest" purely from their own experience in the environment.
The Self-Supervised Attention Module
The core innovation is a self-supervised module that learns to identify foreground objects without human labels. The module utilizes an auto-encoder architecture designed around a specific reconstruction bottleneck.
As shown in Figure 1, the module takes two frames from a trajectory: a source frame ($x_s$) and a target frame ($x_t$). It attempts to reconstruct the target frame by combining the "foreground" of the target with the "background" of the source. A Mask Generator produces an attention mask ($\Psi$) that acts as a gate, identifying pixels belonging to regions of interest. To ensure the model doesn't simply label the entire screen as foreground, a sparsity penalty is applied, forcing the mask to be as compact as possible while maintaining reconstruction accuracy.
Once trained, this mask generator can be "plugged" into standard RL architectures like A2C or ACKTR. The agent's feature maps are multiplied by the attention mask, effectively "muting" background noise and allowing the policy network to focus on relevant objects.
Experimental Results
The researchers evaluated the module using the Atari Arcade Learning Environment (ALE). The method was tested across a diverse set of games, including Carnival, Freeway, Kangaroo, Ms. Pacman, Skiing, and TimePilot.
As illustrated in Figure 3, agents equipped with the self-supervised attention module consistently reached higher test scores and achieved faster convergence than the baseline A2C agents. Beyond performance scores, the method demonstrated superior object detection capabilities, achieving higher precision and recall than state-of-the-art models like Transporter when extracting object keypoints.
Key Insights and Generalization
- Attention as Inductive Bias: By using attention to guide the learning process rather than just explaining it post-hoc, the agent develops a more robust understanding of the environment.
- Class-Agnostic Learning: Because the module is trained on reconstruction, it learns to identify "things that move or stand out" regardless of their specific semantic class (e.g., ghosts vs. pellets).
- Transferability: A significant finding is that attention masks are often transferable. A "universal" mask trained across multiple games can successfully guide an agent in a new, unseen game, suggesting that the concept of "an object" is visually consistent across different domains.
Conclusion
This work demonstrates that self-supervised visual attention is a powerful inductive bias for RL. By filtering out irrelevant background information, agents can learn more efficiently and generalize better across tasks. This approach moves RL closer to human-like perception and provides a foundation for future work in symbolic RL, where agents might reason over collections of discrete objects rather than raw pixel grids.
Summary updated