Introduction
Deep Reinforcement Learning (DRL) agents are notoriously "brittle." While they can master complex tasks, a tiny amount of observation noise or a calculated adversarial "nudge" to their sensors can cause catastrophic failure. This vulnerability, known as the state adversary problem, is a major hurdle for deploying RL in safety-critical systems like autonomous vehicles.
Existing solutions often fall into two camps: computationally expensive adversarial training or smoothness regularization that degrades peak performance. This paper introduces SortRL, a framework designed to be robust, efficient, and high-performing by baking robustness directly into the policy's architecture.
Methodology
SortRL introduces three core innovations to achieve structural robustness:
1. SortNet Architecture
Instead of standard neural networks, the authors use SortNet, an architecture designed to be globally 1-Lipschitz with respect to the $l_\infty$ norm. Mathematically, this guarantees that a small change in input (observation) cannot result in a disproportionately large change in output (action scores).
2. Robust Margin Optimization
For discrete actions, robustness depends on the output margin—the gap between the highest-scoring action and the runner-up. If this gap is sufficiently large, no small perturbation can flip the agent's decision. SortRL explicitly optimizes this margin using a Hinge-loss-based robust regularizer ($L_{Rob}$).
3. Policy Distillation (PD)
Training Lipschitz networks from scratch can be unstable. SortRL uses a Policy Distillation pipeline where a standard "teacher" policy (like PPO or DQN) is first trained. A SortRL "student" is then trained to mimic the teacher's decisions while simultaneously maximizing the robust margin.
Experimental Evaluation
The authors evaluated SortRL across standard benchmarks, including Classic Control (CartPole, Acrobot, MountainCar, LunarLander) and Atari games, testing against adversarial attacks like Projected Gradient Descent (PGD).
As illustrated in the classic control benchmarks, SortRL maintains stable rewards as perturbation strength $\epsilon$ increases, whereas standard PPO performance degrades rapidly once noise is introduced.
Key Results
SortRL consistently outperformed existing state-of-the-art (SOTA) robust RL methods, particularly as attack strength increased:
- Superior Robustness: On the RoadRunner Atari task with a perturbation strength of $\epsilon = 5/255$, SortRL achieved a score of 40,905, outperforming the previous best method (RADIAL) by nearly 30%.
- Certified Performance: On tasks like Pong and Freeway, SortRL achieved an Action Certification Rate (ACR) of 100%, providing a mathematical guarantee that no attack within specified bounds can change the agent's decision.
- Minimal Performance Trade-off: Unlike many robust methods, SortRL maintained near-peak performance in "clean" environments, sometimes even outperforming standard DRL.
Conclusion
SortRL demonstrates that robustness can be treated as a structural property of policy networks. By combining $l_\infty$ Lipschitz continuity with margin optimization and policy distillation, it offers a computationally efficient path to building resilient RL agents without sacrificing optimality. Future work will focus on extending these margin-based principles to continuous action spaces.
Summary updated