Abstract
Deep Reinforcement Learning (RL) agents utilizing visual inputs are highly susceptible to adversarial attacks—small, targeted perturbations that can lead to catastrophic policy failures. This paper introduces BARReL (Bottleneck Attention for Robustness in Reinforcement Learning), a novel defense that embeds Bottleneck Attention Modules (BAM) into standard neural architectures. By utilizing learned spatial and channel attention maps as denoising filters during inference, BARReL effectively prunes adversarial noise from activations. Evaluated on Atari 2600 benchmarks, the method demonstrates significant improvements in reward recovery and action consistency under attack with negligible computational overhead.
Introduction
Vision-based RL agents, while powerful, share the vulnerabilities of standard image classifiers. Tiny, human-imperceptible perturbations to input pixels can drastically alter an agent's decision-making process, posing severe risks in safety-critical applications like autonomous driving or robotics. While adversarial training offers some protection, maintaining high performance during inference remains a challenge. BARReL addresses this by treating attention not just as a feature selector, but as a robust filter for adversarial noise.
Methodology
The BARReL framework enhances standard architectures, such as the Nature-CNN, by integrating a Bottleneck Attention Module (BAM).
Architecture and Attention
The BAM is inserted between early convolutional layers to compute two distinct attention components:
- Spatial Attention: Identifies salient regions within the visual field.
- Channel Attention: Determines the importance of specific feature maps (e.g., edges, textures).
The Recovery Algorithm
The core innovation lies in the inference-time recovery process. When an input is suspected of being compromised, the agent applies its learned attention map to "clean" the feature activations. This is achieved through an element-wise (Hadamard) multiplication between the perturbed activations and the attention map. Because adversarial noise is often distributed across the image, the polarized attention map—which focuses intensely on game-critical objects—naturally suppresses noise in non-salient background regions.
Training Regime
Models are trained using Proximal Policy Optimization (PPO). To ensure the attention maps learn to distinguish between salient features and noise, the authors employ a sparse adversarial training strategy, introducing attacks on only 10% of the training frames.
Experimental Setup
The effectiveness of BARReL was evaluated on three classic Atari 2600 environments: Breakout-v4, SpaceInvaders-v4, and MsPacman-v4. These environments provide diverse visual challenges and temporal dynamics, serving as a rigorous testbed for robustness.
Results and Discussion
The proposed "BAM-CNN-Adv + Recovery" method consistently outperformed standard baselines under adversarial pressure.
Quantitative Performance
In Breakout-v4, the recovery method achieved a reward of 6.3, a 4.2x improvement over the best baseline reward of 1.5 under attack. Furthermore, the "Reversed-TOP-1" metric—measuring how often the cleaned activations restored the original intended action—reached 57.78% for moderate attacks ($\epsilon = 0.01$).
Qualitative Analysis
Visualizations of the activation maps reveal the mechanism's success. While adversarial attacks typically "smear" the activations of standard CNNs, the BARReL-cleaned maps remain sharp and focused on critical game elements like the paddle in Breakout or enemies in Space Invaders.
Conclusion and Future Work
BARReL demonstrates that attention mechanisms can serve as efficient, low-overhead filters for adversarial robustness in RL. While the defense significantly mitigates performance degradation, a gap remains compared to clean-environment performance. Future research will explore the application of BARReL in continuous action spaces, its effectiveness against "white-box" attackers who may attempt to bypass the attention filter, and the integration of more complex mechanisms like Vision Transformers (ViTs).
Summary updated