Spend Attack Compute Only Where It Moves the Gradient: Variance-Guided Adversarial Rollouts for Group-Relative RL

Many recent fine-tuning methods for large models sample a small group of answers per input, score each one, and push the model toward answers that score above the group average. If we also want the model to stay correct when its inputs are adversarially perturbed, the usual recipe is to attack every training input and add a robustness penalty to the reward. That recipe is expensive and also wasteful. When every answer in a group survives the attack, or every answer fails it, the robustness penalty is identical across the group. Its group-relative advantage is then exactly zero, and the costly attack contributed nothing to the update. When the penalty is summed with a noisy task reward before normalization, its signal is drowned out even in the groups where it does vary.

We propose to treat these two problems as one. The statistic that decides how much a reward component counts after normalization, its spread within the group, also predicts in advance whether running an attack on that input is worth paying for. We normalize the task reward, the safety cost and the adversarial-failure cost separately. We give the adversarial term a variance floor so that rare failures neither vanish nor explode. We then route the expensive multi-step attack only to inputs whose estimated chance of producing a mixed group, some answers broken and some not, is high. The test bed is a 3-billion-parameter vision-language driving planner on NAVSIM-v2, where each multi-step image attack costs many extra forward and backward passes through the full model.

Core Claim

Under group-relative policy optimization (GRPO, a critic-free RL method that scores each sampled completion against the mean and standard deviation of its group), a reward component contributes to the update in proportion to its within-group variation. It contributes nothing when it is constant across the group. Two things follow.

First, an adversarial robustness cost must be normalized on its own channel rather than inside a pre-summed reward. Otherwise high-variance task progress suppresses it, which is the same pathology Constrained Group Relative Policy Optimization proved for safety constraints.

Second, the expected informativeness of an attack on input $s$ can be estimated before running it. Model the adversarial outcome of each of the $G$ sampled completions as failing with probability $p_s$. The attack yields a nonzero advantage only if the group is mixed:

$$I(s) = 1 - p_s^{G} - (1-p_s)^{G}.$$

This follows directly from the binomial chance that not all $G$ outcomes agree. We estimate $p_s$ cheaply, from a single-step attack probe plus a running per-scenario failure history. We then spend a fixed multi-step attack budget on the inputs with the highest $I(s)$.

The claim to test is that this allocation matches the robustness of attacking every input at a fraction of the attack compute. It should also clearly beat attacking a random subset of the same size.

Relation to Prior Work

The proposal borrows from three lines of work and departs from each.

From the reward-scale literature it takes the mechanism. CGRPO shows that combining reward and cost before group normalization implicitly reweights them by their variances, and it fixes this with component-wise "scalarized advantages." It evaluates only benign safety constraints, on a gridworld and on NAVSIM-v2. It does not consider adversarial inputs, and it does not address the near-zero-variance groups that rare failures create. The research file lists that second issue among the field's open problems, as "dynamic range drift." Chen et al. (2025) supply motivating evidence of the same washing-out in reasoning models: high-variance task success swamps length penalties.

Conflict-Averse Gradient Aggregation for Constrained Multi-Objective Reinforcement Learning resolves objective conflict by solving a quadratic program over per-objective gradients. That requires one backward pass per objective. Combined with a $K$-step attack, it compounds the cost of adversarial training. For this reason it acts as a constraint on our design rather than a component.

From the adversarial-training literature the proposal takes the cost problem and a warning. Understanding and Improving Fast Adversarial Training and A Training-Dynamics View of Catastrophic Overfitting show that cheap single-step attacks used as the training adversary can produce a model that beats the weak attack while collapsing against multi-step attacks. The second paper also shows such baselines eventually collapse to about 0% robustness under extended training. We therefore use the single-step attack only as a probe for allocation, never as the training adversary. We track their gradient-alignment diagnostic as an early warning sign.

Adversarial Training for Process Reward Models is the nearest work that combines RL rewards with an adversary. It trains a reasoning-error generator against a reward model, reports substantial overhead, and leaves reward-scale interaction unanalysed. Its adversary is a learned generator of fallacious text rather than a perturbation of the input, and it has no mechanism for deciding where adversarial compute goes. Robust Gymnasium motivates the problem by showing that safe-RL methods degrade sharply under disruptions, but it proposes no fix.

Research Program

The method has three parts.

Separate normalization. The total advantage for completion $i$ in a group is computed per component, with a floored standard deviation on the adversarial channel:

$$\hat A_i = \frac{R_i - \mu_R}{\sigma_R} - \lambda_c \frac{C_i - \mu_C}{\sigma_C} - \lambda_a \frac{F_i - \mu_F}{\max(\sigma_F,\ \sigma_{\min})}.$$

Here $R$ is task progress, $C$ is the existing safety cost, and $F$ is the adversarial failure cost (the drop in driving score under attack). The $\mu$ and $\sigma$ terms are within-group means and standard deviations. $\lambda_c$ and $\lambda_a$ are Lagrange multipliers. $\sigma_{\min}$ is a floor set from the running batch-level spread of $F$. The floor keeps the absolute size of a robustness failure meaningful and prevents near-constant continuous costs from amplifying noise.

Allocation. Each training batch has an attack budget $B$, expressed as the fraction of scenarios that receive a full $K$-step attack on the camera inputs. For every scenario we first run a one-step probe, which costs one extra backward pass. We combine the probe outcome with a decayed per-scenario failure history to estimate $p_s$. We rank scenarios by $I(s)$, and the top $B$ fraction receive the full attack on all $G$ rollouts. The remaining scenarios contribute only task and safety advantages. A small random slice of the budget is reserved for exploration, so that scenarios the probe misjudges are still sometimes attacked.

Accounting and diagnostics. All compute is reported in forward-plus-backward passes through the 3B model, including probe cost. Throughout training we log three quantities. The first is the fraction of attacked groups with zero spread in $F$, which measures wasted compute. The second is the effective weight of the adversarial channel under pre-summed normalization, which measures suppression. The third is the input-gradient alignment from the fast-adversarial-training literature, which flags gradient masking.

Baselines

The do-nothing reference is the CGRPO planner trained without any adversarial term, evaluated under the same attacks. For context, CGRPO reports an extended PDM score (EPDMS) of 0.774 in nominal scenarios and 0.516 in out-of-distribution corrective scenarios on NAVSIM-v2 Navhard with Qwen2.5-VL 3B (Constrained Group Relative Policy Optimization). These are clean scores. For EPDMS under adversarial image attack on NAVSIM, no reported result was found on Lacuna, so we measure it ourselves. Any method must stay near CGRPO's clean scores in our reproduction while raising attacked EPDMS.

The zero-cost defense is the "Adversarial Possibility" prompt of Improving Adversarial Robustness in Vision-Language Models with Architecture and Prompt Design. That paper reports LLaVA-7B rising from 12.54% to 91.46% accuracy on COCO under Auto-PGD. We apply the same prompt to the CGRPO planner. It is evaluated under attacks that know the prompt, because a white-box attacker can fold it into the objective.

The naive adversarial RL baseline is standard GRPO on the pre-summed reward $R - \lambda_c C - \lambda_a F$, with every scenario attacked. We sweep $\lambda_a$ over the same grid used for our method.

The strongest competitor is component-wise CGRPO with the adversarial channel added and every scenario attacked with full $K$-step PGD. This is the most direct extension of existing work. It is the quality ceiling we hope to match at lower cost.

The key ablation is component-wise CGRPO with the floored normalization but a uniformly random subset of scenarios attacked, at exactly our budget $B$, including probe-equivalent compute. Beating it isolates allocation as the source of any gain.

A second ablation removes the variance floor. It tests whether the floor matters once allocation already concentrates attacks on mixed groups.

All runs use the same navtrain data, the same group size, the same tuning budget for $\lambda$ and learning rate, and three seeds.

Our method counts as beating the field under two conditions. First, its attacked EPDMS must fall within seed noise of the full-attack competitor while using at most a third of that competitor's attack passes. Second, it must exceed uniform subsampling at matched passes by more than seed noise. If it only ties uniform subsampling, the allocation idea has failed.

Why not just…?

Why not just use a single-step attack everywhere and skip allocation? It is cheaper still. But fast adversarial training shows that models trained against single-step attacks can lose all robustness to iterative attacks. Understanding and Improving Fast Adversarial Training reports FGSM and its random-start variant dropping to about 0% on CIFAR-10 at ε=16/255. The cure in A Training-Dynamics View of Catastrophic Overfitting adds a regularizer whose weight against the task loss is itself a hand-tuned trade-off. We include a single-step-everywhere arm and evaluate it with multi-step attacks. If it holds up on a VLM planner under RL, that would be a real threat, and we would report it as such.

Why not just precompute adversarial camera images offline and fine-tune on them? This removes the attack from the training loop entirely, so it is a serious competitor on cost. Its weakness is that the perturbations go stale as the policy changes, which is the reason online inner maximization exists. We run it at matched total compute. Our method must beat it on attacked EPDMS against attacks computed on the final model.

Why not just raise $\lambda_a$ on the pre-summed reward? CGRPO proves that under pre-combined normalization the effective weight of a low-variance component is set by the combined standard deviation, so it stays suppressed regardless of how large $\lambda$ is. Our $\lambda_a$ sweep and the logged effective-weight diagnostic test whether this holds for adversarial costs.

Why not just add a prompt? It costs nothing, and its reported gains are large. But the gains come from off-the-shelf attacks, and the authors concede that an attacker aware of the prompt can bypass it. If the prompt alone matches our attacked EPDMS under prompt-aware attacks, the training-time method is unnecessary, and the experiment will show this.

Proposed Experiments

Mechanism check. Before the driving runs, we verify the mechanism on the CGRPO 10x10 gridworld with adversarial observation noise. Ground-truth failure probabilities there are computable, so we can check that $I(s)$ ranks scenarios correctly and that allocation reduces zero-spread groups. CGRPO reports reaching its 1% lava-visitation target. We check that adding the adversarial channel preserves that target.

Main study. The main study runs on NAVSIM-v2 Navhard with Qwen2.5-VL 3B. We report nominal and corrective EPDMS clean, and under ℓ∞ Auto-PGD on camera inputs at two radii. For every method we plot attacked EPDMS against total forward-backward passes.

Decisive checks. Three results matter most. The first is whether, under full attack, the fraction of zero-spread adversarial groups is large. If it is small, the waste our allocation targets does not exist, and the idea reduces to plain CGRPO. The second is whether variance-guided allocation beats uniform subsampling at matched passes. The third is whether models trained with probe-guided allocation keep their robustness under strong attacks, with no drop in gradient alignment that would signal masking.