Adversarial Reinforcement Learning as a POMDP

This direction is merged into another direction: input perturbation sensitivity. Its last version stays readable and citable here.

Version 1 of 120 papers38 conceptsfrozen 2026-10-04

In standard Reinforcement Learning (RL), we assume the agent has a direct line to the truth. When an agent receives an observation, it treats that data as the absolute state of the environment. This "Markovian" assumption—that the current observation contains all the information needed to act—is the foundation of most modern RL algorithms.

However, Huan Zhang and Hongge Chen have demonstrated that this assumption makes agents incredibly fragile. By applying tiny, imperceptible perturbations to an agent's input, an adversary can trick a high-performing policy into catastrophic failure. This vulnerability exists because the agent "trusts" its eyes too much.

A growing body of research suggests that the solution isn't just to make the agent's neural network "smoother." Instead, we must fundamentally reframe the problem: Adversarial attacks on observations transform a fully observable environment into a partially observable one.

The Adversary as a Mask

When an adversary can manipulate what an agent sees, the observation $o$ is no longer equal to the true state $s$. Instead, the observation becomes a noisy, potentially malicious signal that hides the ground truth.

In Optimal Attacks on Reinforcement Learning Policies, Alessio Russo and Alexandre Proutiere argue that this shift from a Markov Decision Process (MDP) to a Partially Observable Markov Decision Process (POMDP) is the core challenge of adversarial RL. In a POMDP, an agent cannot simply react to its current input. It must maintain a belief state—a probability distribution over what the true state might be, given the history of observations and actions.

If an agent is memoryless (reactive), it is easily misled. But if an agent uses memory—such as a Recurrent Neural Network (DRQN)—it becomes naturally more resilient because it can reconcile a suspicious current observation with its past experiences.

Modeling the State-Adversarial MDP

To formalize this, Huan Zhang and colleagues introduced the State-Adversarial MDP (SA-MDP) framework in Robust Deep Reinforcement Learning against Adversarial Perturbations on State Observations.

In an SA-MDP, the environment transitions normally, but the agent receives a perturbed observation $\hat{s}$ that is within some bounded distance of the true state $s$. This framework allows researchers to treat robustness as a reachability problem: can we find a policy that performs well for any possible observation the adversary might show us within that bound?

This led to the development of ATLA (Alternating Training with Learned Adversaries), described in Robust Reinforcement Learning on State Observations with Learned Optimal Adversary. Instead of training against random noise, ATLA trains the agent against a "master" adversary that is specifically optimized to find the agent's weakest points.

From Robustness to State Estimation

If adversarial RL is a POMDP, then the agent's job is not just to act, but to infer.

Arunesh Sinha, Roman Belaire, and Pradeep Varakantham take this further with the concept of Adversarial Counterfactual Error (ACoE). In their work, On Minimizing Adversarial Counterfactual Error in Adversarial RL, they treat adversarial noise as a mask hiding the true state.

Their approach, implemented in algorithms like A2B and A3B, explicitly models the agent's belief about the true state. By minimizing the "counterfactual error"—the difference between how the agent acts on a perturbed state versus how it would have acted on the true state—they create agents that can ignore malicious "hallucinations" and stay focused on the underlying environment dynamics.

The Cost of Uncertainty

While treating adversarial RL as a POMDP provides a path to robustness, it comes with a steep computational price. Jeremy McMahan and Xiaojin Zhu point out in Optimal Attack and Defense for Reinforcement Learning that finding an optimal defense against observation attacks is NP-hard.

This hardness stems directly from partial observability. In a standard MDP, the optimal policy is a simple mapping from states to actions. In a POMDP (or an adversarial observation setting), the agent must reason about an infinite space of possible belief distributions.

Furthermore, as Eline M. Bovy and Nils Jansen explore in Imprecise Probabilities Meet Partial Observability, an adversary can manipulate these belief states over time. By strategically changing environmental parameters, an attacker can "steer" the agent's internal estimate of reality, making the agent's own state-estimation process a liability.

Summary of the Intuition

The shift in this research area can be summarized as follows:

  1. The Problem: Adversaries break the Markov assumption by decoupling observations from states.
  2. The Framing: This decoupling turns the environment into a POMDP for the agent.
  3. The Defense: Robustness is achieved not through simple smoothing, but through state estimation. The agent must maintain a belief about the true state and minimize the error between its perceived actions and its "counterfactual" optimal actions.
  4. The Trade-off: While belief-based agents are more robust, the introduction of partial observability makes finding the "perfect" defense computationally difficult.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_adversarial_reinforcement_learning_as_a__d72617e9,
  title = {Adversarial Reinforcement Learning as a POMDP},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/adversarial-reinforcement-learning-as-a-pomdp/rdr_d72617e9c1a849d6b94c00c7789e052f},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_adversarial_reinforcement_learning_as_a__d72617e9_v1,
  title = {Adversarial Reinforcement Learning as a POMDP},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/adversarial-reinforcement-learning-as-a-pomdp/rdr_d72617e9c1a849d6b94c00c7789e052f/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}