Robust Control through Adversarial Game Theory

112 concepts92 papers1 proposal

Standard reinforcement learning (RL) typically optimizes for the average case. An agent learns to maximize its expected reward by interacting with an environment that behaves according to a fixed, often stochastic, distribution. However, this "average-case" optimization creates a significant reality gap: a policy that performs perfectly in a clean simulation can fail catastrophically in the real world, where unmodeled dynamics, sensor noise, or physical disturbances deviate from the training distribution.

To bridge this gap, researchers have turned to a more pessimistic but resilient framework: Robust Reinforcement Learning (RRL). Instead of treating uncertainty as random noise, RRL treats it as an intelligent adversary.

The Minimax Formulation

The core intuition behind this research direction is to frame control as a zero-sum game between two players: a controller (the agent) and a disturber (the environment or an adversary). As Morimoto et al. first proposed, the controller seeks to minimize a cost function while the disturber simultaneously seeks to maximize it.

This creates a minimax optimization problem: $$\min_{\pi} \max_{\delta} J(\pi, \delta)$$ where $\pi$ is the agent's policy and $\delta$ represents the adversarial disturbances. By solving for the saddle-point equilibrium of this game, the agent develops a policy that is guaranteed to perform at a certain level even under the "worst-case" environmental conditions. Ogunmolu et al. demonstrate that this iterative dynamic game approach allows for the quantification of policy brittleness, identifying exactly where a controller might fail.

Bridging the Sim-to-Real Gap

The primary benefit of adversarial training is its ability to generalize to real-world environments that the agent never saw during training. Pinto et al. introduced Robust Adversarial Reinforcement Learning (RARL), showing that training an agent against a destabilizing adversary—one that applies targeted forces to the agent's joints or body—results in policies that are significantly more stable when transferred from simulation to physical robots.

This mechanism works because the adversary identifies and exploits the agent's specific weaknesses. To succeed, the agent must develop a policy that is "smooth" and resilient to those specific perturbations. Aljalbout extended this concept to dual-arm robotics, using one arm as a physical adversary to automate the discovery of failure modes, making real-world learning more scalable.

Safety as a Game

In safety-critical systems like autonomous vehicles, robustness is not just about performance; it is a requirement for survival. Bansal et al. explain how Hamilton-Jacobi (HJ) reachability uses this game-theoretic lens to maintain safety boundaries. By treating environmental noise as a worst-case adversary, the system can calculate a "safe set"—a region of the state space from which the controller can guarantee safety regardless of what the environment does.

Similarly, Yang and Li et al. have developed methods to identify robust invariant sets. These are regions where, even under adversarial disturbances, the system is mathematically guaranteed to remain safe. Almubarak et al. further refined this by embedding "barrier states" directly into the game dynamics, ensuring that safety constraints are never violated during the optimization process.

Beyond Simple Noise: Intelligent Adversaries

While early work focused on physical disturbances, recent research has expanded the definition of the "adversary" to include:

  • Distributional Uncertainty: Taskesen et al. show that linear policies can be optimal even when an adversary chooses the worst-case noise distribution from a set of possibilities.
  • Cyber-Physical Attacks: Lukina and Niu et al. model malicious hackers as adversarial players in a stochastic game, allowing systems to maintain mission objectives even while under active attack.
  • Model Hallucinations: In offline RL, where agents learn from static datasets, Liu et al. use adversarial logic to create a "conservative" Bellman operator. This ensures the agent doesn't trust its own potentially flawed simulations of the environment.

The Cost of Robustness

Training against a nightmare scenario comes with a "safety tax." Iannelli et al. proved that the regret of these robust controllers—the performance lost compared to a perfect, clairvoyant controller—scales directly with the gap between the assumed worst-case disturbance and the actual noise encountered.

To mitigate this, researchers like Lin et al. propose hierarchical structures. A robot might use a natural, efficient gait policy for standard operation but switch to a specialized, adversarial-trained "safety recovery" policy only when extreme disturbances are detected. This allows for high performance in typical conditions without sacrificing the resilience needed for the worst-case scenario.

Key Foundational Work

Robust Reinforcement Learning (2000) — contributed 2 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_adversarial_training_for_robustness_f30dbb3b,
  title = {adversarial training for robustness},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/adversarial-training-for-robustness/rdr_f30dbb3be0f149bfb6cfe9d44ca05a63},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_adversarial_training_for_robustness_f30dbb3b_v1,
  title = {adversarial training for robustness},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/adversarial-training-for-robustness/rdr_f30dbb3be0f149bfb6cfe9d44ca05a63/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake