Adversarial Learning of Distributional Reinforcement Learning

Yang Sui, Yukun Huang, Hongtu Zhu, Fan ZhouICML 2023 · OpenReview

Reinforcement Learning (RL) has a notorious "sim-to-real" problem: a policy that plays a game perfectly in a simulator might fail catastrophically when deployed in the real world. This discrepancy occurs because real-world environments are never identical to simulations—there is always a "gap" in state observations or system dynamics.

Currently, it is difficult to systematically analyze why a model fails. Is it because the agent encountered a specific "fragile" state it didn't recognize? Or is the neural network architecture itself unstable? Researchers have developed a new diagnostic framework for Distributional Reinforcement Learning (DRL) to address the lack of a unified, mathematically rigorous tool to detect these vulnerabilities across different components of an RL system.

A New Diagnostic Framework

The authors introduce a framework based on the First-order Influence Measure (FI), adopting concepts from information geometry to pinpoint exactly which states and neural network parameters are most vulnerable to tiny perturbations.

  • Information Geometry: Instead of just looking at how much a reward changes, the authors treat the DRL model as a "perturbation manifold." They use the Fisher Information Matrix to define a "distance" between the original and perturbed probability distributions of future returns.
  • Invariance: A key advantage of their FI is that it is invariant to how the system is reparameterized. Whether measuring a change in pixels or a change in a latent vector, the FI provides a consistent measure of sensitivity.
  • Distributional Focus: While standard RL looks at expected (average) rewards, this framework specifically targets Distributional RL (DRL), which models the entire distribution of possible returns. This allows for highly granular sensitivity analysis in complex environments.

The framework applies FI to three primary diagnostic tasks: finding fragile states, evaluating architecture stability, and identifying parameter sensitivity.

Key Insights: The "Critical Moment"

When evaluating the framework on the Atari 2600 platform (using games like Breakout, Alien, and Asteroids), researchers discovered the "Critical Moment" insight. In games like Breakout, the vast majority of states have low FI, meaning they are robust to noise. However, a few "fragile states" exhibit massive FI scores.

First-order Influence Measure (FI) across different states, highlighting a few highly fragile moments.

Perturbing these specific high-FI states—even slightly—causes the agent to make a wrong move that eventually leads to failure. Conversely, perturbing low-FI states has zero effect on the final score. Furthermore, in "saturated" environments like Pong and Freeway, where the agent has essentially mastered the game, the FI distribution remains very flat and low (~0.03). This confirms that as a model reaches its reward ceiling, its fragility decreases.

State Sensitivity and Visualizing Vulnerability

The framework allows researchers to visually and quantitatively isolate these vulnerabilities. For example, in Breakout, perturbing a high-FI state (Step 1355) changed the optimal action from "FIRE" to "RIGHT," leading to an 11% drop in the final score. Perturbing a low-FI state with the exact same amount of noise resulted in no change to the trajectory.

Comparison of gradients and perturbations for a high-FI state (1355) versus a low-FI state (1414).

Architecture and Parameter Sensitivity

Beyond individual states, the FI framework reveals structural weaknesses within the RL model itself:

  • Decision Layers are the Weak Link: The analysis showed that early convolutional layers (which handle feature extraction) have very low FI (~0.06). In contrast, the final dense layers (which handle the final decision-making) have FI values over 230. This suggests that the "brittleness" of RL models lives in the decision-making head, not the visual processing.
  • The Positive Parameter Coincidence: In the final layer of the policy network, the authors found that the vast majority of parameters were negative. Interestingly, the few parameters that were positive happened to be the ones with the highest FI. Re-scaling these few positive weights by just 10% could collapse the entire policy (resulting in near-zero rewards), while zeroing out low-FI parameters did nothing.

Limitations and Future Work

While this framework provides a powerful diagnostic lens, the current study focuses on environments with discrete action spaces (like Atari). Expanding this to continuous action spaces, which are more common in robotics, remains a necessary next step. Additionally, while the paper covers state and parameter perturbations, it does not yet deeply explore perturbations in the reward function or the transition dynamics.

Ultimately, this research focuses on detection. The next logical step is using these FI scores to guide proactive defense—specifically, using data augmentation to train models to be more robust on the high-FI "fragile" states identified by the framework.

Summary updated

Abstract

Reinforcement learning (RL) has made significant advancements in artificial intelligence. However, its real-world applications are limited due to differences between simulated environments and the actual world. Consequently, it is crucial to systematically analyze how each component of the RL system can affect the final model performance. In this study, we propose an adversarial learning framework for distributional reinforcement learning, which adopts the concept of influence measure from the statistics community. This framework enables us to detect performance loss caused by either the internal policy structure or the external state observation. The proposed influence measure is based on information geometry and has desirable properties of invariance. We demonstrate that the influence measure is useful for three diagnostic tasks: identifying fragile states in trajectories, determining the instability of the policy architecture, and pinpointing anomalously sensitive policy parameters.
BibTeX
@inproceedings{sui2023adversarial,
  author = {Yang Sui and Yukun Huang and Hongtu Zhu and Fan Zhou},
  title = {{Adversarial Learning of Distributional Reinforcement Learning}},
  booktitle = {International Conference on Machine Learning},
  year = {2023},
  url = {https://openreview.net/forum?id=E8u45VW6Nj},
}
Figure