Understanding and Diagnosing Deep Reinforcement Learning

Ezgi Korkmaz

Abstract

Deep Reinforcement Learning (DRL) has demonstrated remarkable success in complex environments, yet its susceptibility to adversarial perturbations and distributional shifts remains a significant barrier to deployment in safety-critical systems. This paper introduces RA-NLD (Robustness Analysis via Non-Lipschitz Directions), a diagnostic framework designed to probe the decision boundaries of DRL policies. By identifying the most vulnerable directions in a policy's state space, RA-NLD reveals that current "robust" training methods often fail to smooth the decision manifold, instead creating "spikier" and more disjointed patterns of vulnerability. This work provides a methodology for precisely diagnosing instability rather than blindly applying mitigation techniques.

1. Introduction

Deep Reinforcement Learning policies are frequently treated as "black boxes." While they can master Atari games or control autonomous vehicles, they are notoriously sensitive to tiny, often invisible, changes to input pixels—known as adversarial perturbations—and natural distributional shifts like lighting changes or blur.

The core challenge in DRL safety is the lack of tools to precisely diagnose why a policy fails or where its decision boundaries are most fragile. Current robust training techniques, such as adversarial training, are often applied without a clear understanding of their impact on the underlying neural manifold. This research aims to bridge this gap by identifying the specific high-dimensional directions where a policy is most likely to break.

2. Methodology: The RA-NLD Framework

The authors propose RA-NLD, a multi-faceted approach to robustness analysis:

  • Identifying Non-Lipschitz Directions: The algorithm searches for directions in the input space where a minimal change ($\epsilon$) results in the maximum divergence in the policy's action distribution. These represent the "steepest cliffs" on the decision manifold.
  • Spectral Characterization: To move beyond single-state analysis, the framework aggregates these sensitive directions across an entire episode. By calculating the principal eigenvector of these directions, the authors identify a Principal Non-Lipschitz Direction ($G_S$), representing a global signature of vulnerability.
  • Feature Correlation Quotient ($\Lambda$): This metric quantifies the shift in sensitive directions when a policy is subjected to environmental changes, such as attacks or noise.
  • Fourier Analysis: Spatial frequency transforms are applied to sensitivity maps to determine if vulnerabilities are high-frequency (sharp and local) or low-frequency (broad and global).

3. Experimental Setup

The study utilizes the Arcade Learning Environment (ALE), evaluating policies on Pong, BankHeist, RoadRunner, and Freeway. These policies are tested against a portfolio of adversarial attacks (e.g., Carlini & Wagner, Elastic-Net) and natural shifts, including brightness, contrast, rotation, blur, and JPEG compression artifacts.

4. Key Insights and Results

4.1 The Paradox of Robust Training

A significant finding of this study is that "certified robust" training (such as SA-DQN) does not necessarily smooth the decision manifold. Instead, it often results in a "spikier" and more disjointed representation. While these policies may resist specific known attacks, they become more volatile in other directions. This is evidenced by the finding that robustly trained policies exhibit much higher variance in their gradient norms over time compared to standard vanilla policies.

4.2 Quantifying Policy Remolding

Using the Feature Correlation Quotient ($\Lambda$), the authors demonstrated that adversarial attacks significantly "remold" a policy's sensitive directions. In BankHeist, the correlation between a clean policy and an attacked one dropped to 0.17, indicating a near-total shift in the network's internal logic and sensitivity patterns.

4.3 Frequency Signatures

Adversarial training leaves a distinct signature in the spatial frequency domain. Vanilla policies tend to have concentrated frequency vulnerabilities, whereas robust policies spread their sensitivities more evenly across the Fourier spectrum. This suggests that robust training fundamentally changes the "texture" of how the agent perceives the environment.

4.4 Natural vs. Adversarial Shifts

The research highlights a critical gap in current safety research: "natural" shifts, such as JPEG compression, can sometimes be more damaging to a policy's internal representations than intentional adversarial attacks. This suggests that optimizing for adversarial robustness alone is insufficient for real-world reliability.

5. Discussion and Limitations

While RA-NLD provides a powerful diagnostic tool, several limitations remain. The current experiments focused on discrete action spaces (Atari), and further research is required to determine how these findings scale to continuous control tasks like robotics. Additionally, the method currently requires "white-box" access to the policy's gradients. Future work should explore black-box diagnostic versions and seek to reduce the computational overhead associated with calculating eigenvectors across high-dimensional state sets.

6. Conclusion

RA-NLD moves the field of DRL safety from reactive mitigation to proactive diagnosis. By visualizing and quantifying the "why" behind policy failure, researchers can better understand the trade-offs of robust training and develop agents that are truly resilient to both adversarial intent and natural environmental variance.

Summary updated

Abstract

Deep neural policies have recently been installed in a diverse range of settings, from biotechnology to automated financial systems. However, the utilization of deep neural networks to approximate the value function leads to concerns on the decision boundary stability, in particular, with regard to the sensitivity of policy decision making to indiscernible, non-robust features due to highly non-convex and complex deep neural manifolds. These concerns constitute an obstruction to understanding the reasoning made by deep neural policies, and their foundational limitations. Hence, it is crucial to develop techniques that aim to understand the sensitivities in the learnt representations of neural network policies. To achieve this we introduce a theoretically founded method that provides a systematic analysis of the unstable directions in the deep neural policy decision boundary across both time and space. Through experiments in the Arcade Learning Environment (ALE), we demonstrate the effectiveness of our technique for identifying correlated directions of instability, and for measuring how sample shifts remold the set of sensitive directions in the neural policy landscape. Most importantly, we demonstrate that state-of-the-art robust training techniques yield learning of disjoint unstable directions, with dramatically larger oscillations over time, when compared to standard training. We believe our results reveal the fundamental properties of the decision process made by reinforcement learning policies, and can help in constructing reliable and robust deep neural policies.
BibTeX
@inproceedings{korkmaz2024understanding,
  author = {Ezgi Korkmaz},
  title = {{Understanding and Diagnosing Deep Reinforcement Learning}},
  booktitle = {International Conference on Machine Learning},
  year = {2024},
  url = {https://openreview.net/forum?id=s9RKqT7jVM},
}
Figure