Adversarial Perturbations on Reinforcement Learning Observations

727 concepts335 papers1 proposal

Deep reinforcement learning (RL) agents have achieved remarkable success in complex tasks, from mastering video games to controlling autonomous vehicles. However, these high-performing policies possess a critical vulnerability: they are notoriously fragile when faced with minor, often human-imperceptible perturbations to their state observations.

Understanding why this fragility exists—and how to mathematically defend against it—requires looking closely at how sequential decision-making differs from static classification.

The Cascading Fragility of Sequential Decisions

In standard computer vision, an adversarial perturbation causes a one-off misclassification (e.g., mistaking a stop sign for a speed limit sign). If the next frame is clean, the model recovers. In reinforcement learning, however, the agent operates in a closed loop where actions dictate future states.

When an adversary subtly manipulates an agent's observation, the agent executes a suboptimal action. This action drives the environment into a state the agent might not have encountered during training. As the loop continues, these minor errors compound, causing the agent's trajectory to diverge entirely from the optimal path.

Sandy Huang et al. demonstrated this test-time vulnerability in Adversarial Attacks on Neural Network Policies by computing the gradient of the policy's loss function with respect to the input state. By adding a tiny, strategically placed perturbation in the direction of maximum loss, they showed that state-of-the-art agents could be induced to fail catastrophically. Even superhuman models are not immune; Cho-Jui Hsieh et al. showed in Are AlphaZero-like Agents Robust to Adversarial Perturbations? that adding semantically meaningless elements to a game board can trigger complete logical failures in elite game-playing agents.

Why RL Policies Have Thin Margins

To understand why these perturbations are so effective, we must look at the underlying optimization objectives of deep RL.

In supervised learning, cross-entropy loss actively pushes the model to maximize the margin between the correct class and the runner-up. By contrast, as Yanchao Sun et al. explain in Who Is the Strongest Enemy? Towards Optimal and Efficient Evasion Attacks in Deep RL, standard RL algorithms focus on minimizing Bellman error. They do not inherently maximize the action margin (the difference in value between the best action and the second-best action).

As a result, the decision boundaries mapping observations to actions are often razor-thin. This structural weakness can be quantified through the lens of Jacobian conditioning. As explored in Exploring and Exploiting Conditioning of Reinforcement Learning Agents, poorly conditioned policy mappings mean that a minuscule shift in the input observation space yields a massive, erratic swing in the output action space.

Formalizing the Threat via State-Adversarial MDPs

To design rigorous defenses, researchers needed a way to model this vulnerability mathematically. Huan Zhang et al. established this foundation in Robust Deep Reinforcement Learning against Adversarial Perturbations on State Observations by introducing the State-Adversarial Markov Decision Process (SA-MDP).

In a standard MDP, an agent at state $s_t$ receives the true observation of that state. In an SA-MDP, the true underlying environment state remains $s_t$, but the agent receives a corrupted observation:

$$\hat{s}_t = \nu(s_t)$$

where $\nu$ represents an adversary operating under a bounded budget $\epsilon$, such that $\|\hat{s}_t - s_t\| \le \epsilon$. This framework explicitly decouples the true environmental dynamics from the agent's perception, allowing researchers to isolate and study the robustness of the perception-to-action mapping.

Standard MDP:       State (s_t) ───────────────────────────► Policy (π) ──► Action (a_t)
                                                              ▲
State-Adversarial:  State (s_t) ──► Adversary (ν) ──► Obs (ŝ_t) ┘

Defending Policies: From Min-Max Optimization to Certified Smoothing

Armed with the SA-MDP framework, the research community has developed several distinct paradigms to build robust policies.

Alternating and Curriculum Training

The most direct defense is to train the agent against the worst-case scenario using a min-max formulation. Huan Zhang et al. developed this in Robust Reinforcement Learning on State Observations with Learned Optimal Adversary via the ATLA (Alternating Training with Learned Adversaries) framework. ATLA trains a separate network to find the most damaging observation perturbations while simultaneously training the policy to resist them.

Because an optimal adversary can easily overwhelm an agent early in training, Junlin Wu et al. introduced a curriculum-based approach in Robust Deep Reinforcement Learning through Bootstrapped Opportunistic Curriculum, which progressively scales up the adversary's perturbation budget as the agent's policy matures.

Worst-Case Bellman Operators

Training a dedicated adversary adds massive computational overhead. To circumvent this, Furong Huang et al. proposed WocaR-RL in Efficient Adversarial Training without Attacking: Worst-Case-Aware Robust Reinforcement Learning. Instead of generating explicit attacks, WocaR-RL modifies the Bellman operator itself to directly calculate and optimize for the lower bound of rewards within an $\epsilon$-ball neighborhood of the state, embedding worst-case awareness directly into the value function updates.

Provable Robustness via Randomized Smoothing

While empirical defenses improve resilience, they offer no mathematical guarantees against a novel, highly adaptive attacker. To achieve formal guarantees, researchers adapted randomized smoothing from supervised learning to sequential decision-making.

Aounon Kumar et al. introduced this paradigm in Policy Smoothing for Provably Robust Reinforcement Learning, alongside parallel work by Fan Wu et al. in CROP: Certifying Robust Policies for Reinforcement Learning through Functional Smoothing.

The intuition behind policy smoothing is elegant: rather than evaluating the policy on a single point observation, the agent samples many versions of the observation corrupted by random Gaussian noise. By aggregating the policy's actions over this noisy distribution (e.g., via majority voting), the sharp, thin decision boundaries of the neural network are smoothed out.

This allows the authors to mathematically certify a "robust radius." Within this radius, no matter how strategically an adversary constructs a perturbation, the smoothed policy is guaranteed to maintain its action selection or preserve a lower bound on its total episodic reward.

Key Foundational Work

Adversarial Attacks on Neural Network Policies (2017) — contributed 7 concept elements to this direction.
Adversarial Attacks on Neural Network Policies (2017) — contributed 7 concept elements to this direction.
Reinforcement Learning for Autonomous Defence in Software-Defined Networking (2018) — contributed 6 concept elements to this direction.
History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 360 papers
  2. Version 1 · 2026-10-04 · 246 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_input_perturbation_sensitivity_0f8be910,
  title = {input perturbation sensitivity},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/input-perturbation-sensitivity/rdr_0f8be9105a5a4f12b06459dae5f3a2e7},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 2, frozen text):

@misc{lacuna_input_perturbation_sensitivity_0f8be910_v2,
  title = {input perturbation sensitivity},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/input-perturbation-sensitivity/rdr_0f8be9105a5a4f12b06459dae5f3a2e7/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake