Attacking RL Policies by Forking the GPU Simulator

Claims that a reinforcement learning (RL) policy is robust rest on proxy attacks. A reinforcement learning policy is a network that maps what an agent observes to what it does. Robustness here means it keeps performing when an adversary slightly corrupts those observations. The standard attacks never measure what the adversary actually wants, which is lower future reward. Instead they push the policy's chosen action away from its clean choice, or they lower a learned value estimate, one step at a time. This is a convenience forced by hardware. When the game or physics engine runs on the CPU, you cannot cheaply ask "what happens over the next 30 steps if I apply this perturbation now?" Answering would mean copying the environment state, replaying many futures, and shuttling observations back and forth to the GPU for every candidate.

Simulators that run entirely in GPU memory remove that constraint. With them, the environment state can be cloned thousands of times and stepped forward in parallel next to the policy. We propose a forking attack: at each step, gradient methods propose many candidate perturbations inside the allowed budget. The simulator then plays each candidate forward for a short horizon in parallel copies, and the attacker applies the one that actually loses the most return. The central question is whether published robust policies stay robust when the adversary optimizes the true objective rather than a proxy. We think some of them will not, and the ways they fail will show what those defenses really learned.

Core Claim

Our hypothesis is that one-step proxy attacks overestimate robustness in sequential control. Harm in RL is delayed and depends on where the attack lands. A perturbation that barely changes the current action can still send the agent into a state it cannot recover from, and a proxy attack cannot see that. Policy Smoothing for Provably Robust Reinforcement Learning shows formally that adaptive adversaries gain by concentrating their budget at high-leverage steps. That paper motivates this work. Existing empirical attacks do not search for those steps using actual outcomes.

The new element is using the simulator itself as part of the attack's objective. We cloned and rolled forward environment copies are used to score and select perturbations. This is only affordable when simulation shares GPU memory with the policy. It turns the throughput of engines like CuLE, which reports up to 155M frames per hour on one GPU, into attack strength instead of training speed.

The claim is falsifiable. The forking attack could fail to beat the strongest proxy attacks from SA-MDP at equal perturbation budget. If so, proxy attacks are adequate, and that is a useful negative result for the field's evaluation practice.

Research Program

How the attack works

The attacker takes the observation ball $B_\epsilon(s_t)$, the set of perturbed observations within $\ell_\infty$ distance $\epsilon$ of the true observation $s_t$. It seeds $N$ candidates from this ball: a few from projected gradient descent (PGD, repeated small gradient steps projected back into the ball) on the SA-MDP proxy losses, and the rest sampled at random within the ball. For each candidate $\delta_i$, it clones the simulator state. It then rolls the victim policy forward $H$ steps with $\delta_i$ applied at step $t$, plus whatever default attack runs afterward. It picks

$$\delta_t^\star = \arg\min_{i \le N} \; \hat{R}_H(s_t, \delta_i),$$

where $\hat{R}_H$ is the realized $H$-step return of the fork. For long-horizon credit, a bootstrapped value can be added at the fork's end.

Evaluating every step this way costs too much, so a cheap screen decides where to fork. The screen is the victim's own value gap under the one-step attack. This gives the attack a way to find the high-leverage steps predicted by Policy Smoothing. We also run a variant with a total episodic budget, the threat model from that paper, in which the attacker spends its budget only on steps where forking predicts large harm.

Systems work

The engineering work is to keep forks, policy inference and candidate generation all in GPU memory. On Atari we use CuLE and its cache-based parallel resets. On continuous control we use Isaac Gym and Isaac Lab.

Forking helps with a known weakness of CuLE. That paper reports efficiency degrading sharply below about 128 parallel instances. A forking attack naturally creates thousands of instances from one evaluation episode, so it runs in the regime where GPU emulation performs best.

We also measure the fork cost against a CPU fork built on EnvPool with MuJoCo state save and restore. This answers directly the question of whether GPU residency is necessary.

Analysis beyond attack strength

We characterize where the forking attack strikes: which game or physics phases, and how far ahead the harm shows up. We compare its chosen steps with those an action-difference attack would choose.

We also check every result against a certificate. For policies trained with Gaussian policy smoothing, the certified lower bound on episodic reward from Policy Smoothing must never exceed the return achieved under our attack. That gives a correctness test for the attack.

The same parallel simulation lets us compute those certificates with far more than the 10,000 episodes per point that paper used. That in turn lets us see how quickly the certificates become vacuous as the horizon grows, compared with how quickly the empirical attack's advantage grows.

Baselines

Naive reference points. The first reference is clean return with no attack. The second is uniform random noise at the same $\epsilon$. Together they set the floor any attack must beat and show how much a budget-matched but unstructured perturbation already costs.

Strongest existing attacks. These are the attacks from Robust Deep Reinforcement Learning against Adversarial Perturbations on State Observations. They are the Maximal Action Difference attack and the Robust Sarsa attack for PPO, and PGD on Q-values for DQN. They are the strongest competitors because they are the attacks against which current robustness claims were made.

The file reports these reference figures from that paper:

Setting Clean return Under attack Robust policy under attack
Hopper, vanilla PPO vs SA-PPO, MAD attack ~3167 ~733 ~1224 (SA-PPO)
Pong, vanilla DQN vs SA-DQN, PGD attack winning score −21 +21 (SA-DQN)

We rerun these attacks ourselves on the same victim checkpoints, observation normalization and $\epsilon$. For fairness we give them extra compute: PGD steps and random restarts are raised until they match the forking attack's wall-clock per episode, and the hyperparameter search budget is the same. Under that matched budget, the forking attack beats them if it lowers mean episodic return significantly across seeds. The decisive case is lowering SA-DQN on Pong well below +21, or SA-PPO on Hopper below the ~1224 reported under MAD. Both would be at the same $\epsilon$.

Victims. The strongest existing defenses serve as victims. They are SA-PPO and SA-DQN from the same paper and Gaussian-smoothed policies from Policy Smoothing. Each is trained with the authors' published objectives in our GPU simulators and alongside vanilla PPO and DQN.

Ablations that isolate the key ingredient. Four ablations separate what matters.

  1. Horizon $H = 0$. This keeps the same $N$ candidates but scores them with the one-step proxy. If it matches the full attack, forward simulation adds nothing.
  2. Value-function scoring. This replaces rollouts with the victim's own value estimate at the fork. It separates using the true dynamics from simply looking ahead through a learned model.
  3. Uniform step selection. This removes the leverage screen and forks at uniformly chosen steps under the same total budget. It tests whether finding the right steps matters.
  4. Gradient seeds only. This drops the random candidates.

Why not just…?

Why not just run PGD with more steps and restarts on the existing proxy losses? This is the cheapest serious alternative. Our compute-matched baseline tests it directly. If stronger proxy attacks close the gap, our claim fails. Policy Smoothing's result that adaptive adversaries gain by timing their budget suggests they will not close it. A per-step proxy has no signal about timing, however many steps it takes. The $H = 0$ ablation separates these explanations cleanly.

Why not just fork a CPU simulator with EnvPool? This is a real threat for low-dimensional MuJoCo, where state save and restore is cheap. EnvPool reaches 3 million frames per second on a DGX-A100. We therefore build that CPU-fork pipeline and report attack strength per wall-clock hour for both.

We expect the GPU advantage to be large for pixel observations. There, every one of the $N \times H$ fork steps would otherwise send a frame across the bus for policy inference. We expect it to be modest for Hopper. We will report whichever regime the data show, and restrict the systems claim to that regime.

Why not just learn an adversary policy, or rely on certificates instead of attacks? Certificates are attack-independent. Yet the file notes that Policy Smoothing certificates become vacuous over full-length Pong episodes, so they cannot tell apart defenses that differ on long horizons. We use them as a consistency check, not a substitute.

A learned adversary network is a reasonable alternative. However, it approximates exactly the return-minimizing search that forking performs by brute force. Forking results give an estimate of the room left over that tells us whether training such an adversary is worthwhile.

Proposed Experiments

Atari. We use Pong and Freeway on CuLE. These are the games where SA-DQN and Policy Smoothing were evaluated. We test at the $\epsilon$ values used in SA-MDP and report mean episodic return under each attack across at least five victim seeds.

Continuous control. We use Hopper, Walker2d and Humanoid in two setups. The first is CPU MuJoCo via EnvPool, which keeps comparability with the published SA-PPO numbers. The second is the nearest Isaac Gym equivalents, for throughput scaling. For these tasks we report return under attack, attack cost in simulator frames and wall-clock time, and the gap between attack return and certified bound.

Decisive checks. Four results decide the outcome.

  1. Does SA-DQN's +21 on Pong survive the forking attack at the SA-MDP budget? A large drop means published robustness was partly an artifact of the attack.
  2. Does return under attack fall steadily as $H$ grows from 0? A steady fall supports the delayed-harm mechanism. A flat curve refutes it.
  3. Does the leverage screen concentrate attacks on a small fraction of steps while matching full-coverage harm? That would give empirical support to the budget-timing argument of Policy Smoothing.
  4. Do rankings among defenses change under the forking attack compared with MAD and PGD? If the ranking is stable, proxy attacks remain useful for comparing defenses even if they misstate absolute robustness. That is the most likely partial-failure outcome, and it is still worth reporting.