Unsupervised Salient Patch Selection for Data-Efficient Reinforcement Learning

Zhaohui Jiang, Paul Weng

TL;DR

SPIRL (Salient Patch Input Reinforcement Learning) improves the data efficiency of vision-based RL by using a pre-trained Masked Auto-Encoder to identify and process only the "salient" (important) parts of an image. By focusing on patches that are hard to reconstruct from their neighbors—typically moving objects or game-critical elements—the agent learns faster and requires significantly less pre-training time than existing state-of-the-art methods.

What Problem Does This Paper Address?

Deep Reinforcement Learning (RL) is notoriously "data-hungry." To learn to play a simple video game from pixels, an agent often needs to experience millions of frames. This happens for two main reasons:

  1. Learning from Scratch: Most agents start with zero knowledge of visual features, learning to "see" and "act" simultaneously.
  2. Visual Redundancy: A typical game frame is mostly static background. Forcing an agent to process every pixel in every frame is computationally wasteful and introduces noise that slows down learning.

While previous "keypoint-based" methods tried to solve this by identifying important dots in an image, they often require a fixed number of points to be defined beforehand and can be incredibly slow to pre-train.

Methods

The authors introduce SPIRL, which consists of three main components:

  1. Adapted Masked Auto-Encoder (MAE): The researchers pre-train a small Vision Transformer (ViT) based MAE on frames collected from a random policy. Unlike standard MAEs, they use a tiny encoder and a larger decoder. This forces the encoder to focus only on essential local details, while the decoder learns to "memorize" the static background in its weights.
  2. Salient Patch Selection (Dynamic K): To find what matters, SPIRL asks the MAE to reconstruct a patch using only its neighbors. If the error is high, the patch is "salient" (likely a moving object or a complex feature). Instead of picking a fixed number of patches, they use a Dynamic K strategy based on the Lorenz curve (a mathematical tool usually used for wealth distribution) to adaptively decide how many patches are needed for a specific frame.
  3. Transformer-based RL Agent: The selected patches are fed into a Transformer-based architecture. Because Transformers can handle variable-length inputs, the agent can process a different number of salient patches at every timestep. This is then integrated with Data-Efficient (DE) Rainbow, a high-performance RL algorithm.

Datasets

The method was evaluated on the Atari 2600 benchmark (via the Arcade Learning Environment), specifically focusing on four games with diverse properties:

  • MsPacman and Frostbite (complex movement and goals).
  • Seaquest (many small moving objects).
  • BattleZone (first-person 3D-like environment).

Key Insights and Ideas

  • The Decoder is the Background: By making the decoder larger than the encoder, the authors found that the decoder effectively learns to generate the game's static background from almost no input. This leaves the encoder free to focus purely on the "salient" moving parts.
  • Reconstruction Error as Saliency: The "aha moment" here is that if a neural network can't guess what a patch looks like based on the patches around it, that patch probably contains something unique and important (like a bullet, an enemy, or a score counter).
  • Efficiency in Pre-training: SPIRL is remarkably fast. It requires only about 1.6 hours of pre-training, compared to 25–50 hours for previous methods like Transporter or PermaKey.

Key Results

  • Superior Sample Efficiency: In the "low-data regime" (100K interaction steps), SPIRL outperformed or matched state-of-the-art methods like SimPLe and PermaKey across most tested games.
  • Performance Boost: In Frostbite, SPIRL achieved a score of 669.5 compared to DE-Rainbow's 341.4. In Seaquest, it reached 557.9 vs 395.6.
  • Interpretability: Because the model uses an attention mechanism on specific patches, the researchers could visualize exactly what the agent was looking at. The attention maps confirmed the agent focuses on game-critical elements like oxygen bars, enemies, and submarines.

Limitations and Future Work

  • Background Importance: SPIRL's core philosophy is to ignore the background. However, in some games, the background might contain vital clues for decision-making. In MsPacman, for instance, the maze corridors are treated as background, which can slightly hinder performance.
  • Moving Backgrounds: The current method might struggle with games that have complex, constantly scrolling backgrounds, as the reconstruction error might flag the entire screen as "salient."
  • Future Directions: The authors suggest exploring ways to selectively re-incorporate background elements if they are found to be important for the agent's policy, and applying this salient patch selection to other non-RL computer vision tasks.

Summary updated

Abstract

To improve the sample efficiency of vision-based deep reinforcement learning (RL), we propose a novel method, called SPIRL, to automatically extract important patches from input images. Following Masked Auto-Encoders, SPIRL is based on Vision Transformer models pre-trained in a self-supervised fashion to reconstruct images from randomly-sampled patches. These pre-trained models can then be exploited to detect and select salient patches, defined as hard to reconstruct from neighboring patches. In RL, the SPIRL agent processes selected salient patches via an attention module. We empirically validate SPIRL on Atari games to test its data-efficiency against relevant state-of-the-art methods, including some traditional model-based methods and keypoint-based models. In addition, we analyze our model’s interpretability capabilities.
BibTeX
@inproceedings{jiang2023unsupervised,
  author = {Zhaohui Jiang and Paul Weng},
  title = {{Unsupervised Salient Patch Selection for Data-Efficient Reinforcement Learning}},
  booktitle = {ECML/PKDD},
  year = {2023},
  url = {https://openreview.net/forum?id=8HZsT6Cvoj},
}

Loading related content...

Figure