Unsupervised Salient Patch Selection for Data-Efficient Reinforcement Learning

Zhaohui Jiang, Paul Weng

Abstract

Deep Reinforcement Learning (RL) from pixels is notoriously data-inefficient, often requiring millions of frames to learn simple tasks. This inefficiency stems from the simultaneous learning of visual representations and control policies, compounded by the high redundancy of visual inputs. We introduce SPIRL (Salient Patch Input Reinforcement Learning), a method that improves data efficiency by focusing on "salient" image patches. By leveraging a pre-trained Masked Auto-Encoder (MAE), SPIRL identifies patches that are difficult to reconstruct from their neighbors—typically representing moving objects or critical game elements. Our approach significantly reduces pre-training time and achieves competitive performance on the Atari 100k benchmark.

Introduction

The "data hunger" of vision-based RL is a major bottleneck. Standard agents process entire frames, most of which consist of static, task-irrelevant backgrounds. This introduces unnecessary noise and computational overhead. While prior work has explored keypoint-based representations, these often require fixed-size inputs and lengthy pre-training.

SPIRL addresses these challenges by using an unsupervised saliency mechanism to filter visual inputs. By focusing only on the most informative parts of an image, the agent can learn more effectively from fewer interactions.

Methodology

The SPIRL framework consists of three primary components: an adapted Masked Auto-Encoder for saliency detection, a dynamic patch selection strategy, and a Transformer-based RL agent.

Adapted Masked Auto-Encoder (MAE)

We pre-train a small Vision Transformer (ViT) based MAE on frames collected from a random policy. Unlike standard architectures, we employ a tiny encoder and a significantly larger decoder. This design choice forces the encoder to capture only essential local details, while the decoder's capacity allows it to "memorize" the static background of the environment within its weights.

The SPIRL architecture: An MAE identifies salient patches which are then processed by a Transformer-based RL agent.

Salient Patch Selection (Dynamic K)

Saliency is determined by the MAE's reconstruction error. If a patch cannot be accurately predicted from its neighbors, it likely contains a unique, dynamic feature (e.g., the player character or an enemy). Rather than selecting a fixed number of patches, SPIRL uses a Dynamic K strategy. By applying the Lorenz curve to the distribution of reconstruction errors, the model adaptively decides how many patches are necessary to represent the current frame's state.

Transformer-based RL Agent

The selected patches are fed into a Transformer-based architecture integrated with Data-Efficient (DE) Rainbow. The Transformer's inherent ability to handle variable-length sequences makes it ideal for processing the varying number of salient patches identified at each timestep.

Experimental Results

We evaluated SPIRL on the Atari 2600 benchmark, focusing on games with diverse visual characteristics: MsPacman, Frostbite, Seaquest, and BattleZone.

Sample Efficiency and Performance

In the low-data regime (100,000 interaction steps), SPIRL demonstrated superior sample efficiency. For instance, in Frostbite, SPIRL achieved a score of 669.5, nearly doubling the 341.4 score of the baseline DE-Rainbow. Similar gains were observed in Seaquest.

Visualization of saliency detection. The error maps (third column) highlight the most informative regions, such as moving sprites and UI elements, which are then selected for the RL agent.

Pre-training Efficiency

A key advantage of SPIRL is its training speed. While previous state-of-the-art methods like Transporter or PermaKey require 25–50 hours of pre-training, SPIRL reaches convergence in approximately 1.6 hours. This makes it a highly practical solution for rapid experimentation.

Discussion and Limitations

The core philosophy of SPIRL—ignoring the background—is highly effective for games where the background is static. However, this can be a limitation in environments where background elements provide critical context. In MsPacman, for example, the maze walls are often treated as background, which can occasionally hinder the agent's navigation. Additionally, environments with complex, scrolling backgrounds might cause the reconstruction error to flag the entire screen as salient, reducing the benefits of patch selection.

Conclusion

SPIRL offers a robust and efficient path toward data-efficient RL by focusing on salient visual information. By utilizing the reconstruction error of an MAE as a proxy for importance, we can drastically reduce the noise in the agent's input. Future work will explore methods to selectively re-integrate background information when necessary and apply this saliency-driven approach to broader computer vision tasks.

Summary updated

BibTeX
@inproceedings{jiang2023unsupervised,
  author = {Zhaohui Jiang and Paul Weng},
  title = {{Unsupervised Salient Patch Selection for Data-Efficient Reinforcement Learning}},
  booktitle = {ECML/PKDD},
  year = {2023},
  url = {https://openreview.net/forum?id=8HZsT6Cvoj},
}
Figure