CAMP in the Odyssey: Provably Robust Reinforcement Learning with Certified Radius Maximization

Derui Wang, Kristen Moore, Diksha Goel, Minjune Kim, Gang Li, Yang Li, Robin Doss, Minhui Xue, Bo Li, Seyit Camtepe, Liming Zhu

Introduction

Deep Reinforcement Learning (DRL) agents are notoriously brittle. A tiny, well-calculated "nudge" to an agent's observations—whether from sensor noise or a deliberate adversarial attack—can lead to catastrophic failures, such as a robot crashing or a trading bot making disastrous decisions. To address this, researchers have turned to Randomized Smoothing (RS), which provides a "certified radius": a mathematical guarantee that an agent's performance will remain above a certain threshold as long as perturbations stay within a defined "safety bubble."

However, a significant hurdle remains: the trade-off between performance (utility) and provable robustness. Existing methods often rely on simple noise augmentation during training, which does not explicitly optimize for these mathematical guarantees.

The CAMP Framework

The authors introduce CAMP (Certified rAdius-Maximizing Policy), a novel training framework designed to directly optimize the certified radius. Unlike traditional methods, CAMP moves beyond simple data augmentation through three core innovations:

  1. Differentiable Certified Radius: The framework utilizes a differentiable surrogate loss for the certified radius, allowing the neural network to "stretch" its safety bubble using standard gradient descent.
  2. Maximizing the "Q-Gap": A key theoretical insight links the local certified radius to the "Q-gap"—the difference between the Q-value of the optimal action and the runner-up. CAMP explicitly trains the agent to widen this gap, making it significantly harder for an adversary to force a sub-optimal decision.
  3. Policy Imitation for Stability: Directly optimizing for robustness can destabilize RL training. To counter this, CAMP employs a dual-network strategy: a Reference Network (a "stable teacher" trained with standard RL) and a Primary Network that mimics the teacher while incorporating the CAMP robustness loss.
Overview of the CAMP framework, illustrating the training process with policy imitation and the resulting certification benefits.

Key Insights and Results

The most striking "aha moment" of this research is the mathematical connection between the geometric radius of robustness and the internal confidence of the Q-network. By framing robustness as a gap-maximization problem, the authors transform a difficult verification challenge into a standard optimization task.

The empirical results are compelling:

  • Performance Gains: In environments like Cartpole-5, CAMP achieved up to twice the certified expected return of existing state-of-the-art defenses.
  • Universal Robustness: CAMP consistently outperformed baselines (Gaussian Augmentation and NoisyNet) across diverse environments, including autonomous driving simulations (highway-fast-v0) and high-dimensional Atari games (Freeway, Pong, Bank Heist).
  • Empirical Resilience: Beyond theoretical guarantees, CAMP agents proved significantly more resilient to actual adversarial attacks, such as PGD and AutoAttack. In the Highway environment, CAMP agents remained nearly unaffected by attacks that caused baseline agents to fail entirely.
Comparison of certified radius and expected return, showing how CAMP (blue) expands the safety bubble compared to standard methods (red).

Limitations and Future Directions

While CAMP represents a major step forward, it currently focuses on discrete action spaces. Extending the framework to continuous control—essential for many robotics applications—is a primary goal for future work. Additionally, the dual-network training approach introduces a computational overhead of approximately 1.5x to 2x GPU hours compared to standard training. Finally, while effective on Atari, the gains in high-dimensional visual spaces suggest that further refinement of Q-gap optimization for pixel-based inputs is needed.

The authors plan to address these challenges by adapting CAMP for continuous action spaces and optimizing the training pipeline for real-time applications.

Summary updated

BibTeX
@inproceedings{wang2025camp,
  author = {Derui Wang and Kristen Moore and Diksha Goel and Minjune Kim and Gang Li and Yang Li and Robin Doss and Minhui Xue and Bo Li and Seyit Camtepe and Liming Zhu},
  title = {{CAMP in the Odyssey: Provably Robust Reinforcement Learning with Certified Radius Maximization}},
  booktitle = {USENIX Security},
  year = {2025},
  url = {https://openreview.net/forum?id=5svh8o7sHT},
}
Figure