Curvature-Bounded Pessimism for Offline RL Without Adversarial Search

Offline reinforcement learning trains an agent from a fixed log of past experience, with no new interaction. To stop the agent from trusting actions the log never tried, current methods use pessimism: they subtract an uncertainty penalty from the learned value estimate, the network that scores how good an action is in a given state. A known side effect is that this pessimism makes the value function spiky. Small changes to the observed state, such as sensor noise or a deliberate attack, can then swing the value estimate wildly. The existing fixes run an adversary during training that searches for the worst nearby state at every update, and that search is expensive. We propose to drop the search. A second-order Taylor bound gives, in closed form, how far the value can fall inside a small ball of state perturbations. We subtract that bound directly as the pessimism penalty in the learning target and in the policy objective.

The central claim is that robustness to state perturbations can be written as one more pessimism term, the value's worst-case drop within a physical radius $\epsilon$, instead of being added as a separate adversarial-training loop. This term needs only the target network's first-order state gradient plus one finite-difference curvature probe. The expected training overhead is a fraction of that of multi-step adversarial smoothing. We also test a prediction that nobody has checked. Cheap single-step adversarial smoothing of value networks should suffer the same "catastrophic overfitting" seen in fast adversarial training for image classifiers, where the network learns to fool the one-step attacker while staying vulnerable to multi-step attacks. A curvature-aware bound should avoid this failure.

Where the gap is

RORL motivates the work. It shows that point-wise conservatism in CQL and EDAC produces spiky value functions. It fixes this with an ensemble of 10 value networks, an asymmetric smoothing loss evaluated on adversarially sampled perturbed states, and ensemble-disagreement penalties, reaching an average normalized D4RL Gym score of 85.7. The authors themselves flag the cost of the adversarial sampling. Towards Robust Policy supplies both the threat model and the cost figure. Its critic- and actor-smoothing defenses recover about 10 to 11% under attack, but actor defense raises TD3+BC training from 3.05 to 4.55 hours.

Two papers constrain the design space.

  • Gradient Penalty and Constraint Relaxation penalizes the value's gradient with respect to actions for under 10% overhead. It never touches state gradients, so it offers no defense against observation attacks. - Belief-Enriched Pessimistic Q-Learning poses a maximin over a ball of candidate true states, as we do. It resolves that maximin with belief filters or diffusion purification in online settings, which moves the cost to every decision step.

From supervised learning we take the method. Quadratic Upper Bound replaces the inner attack loop with a second-order bound on cross-entropy over logits, and it prevents catastrophic overfitting (AutoAttack accuracy on CIFAR-10 ResNet18 goes from 45.17% to 46.60% on top of N-FGSM). GradAlign shows that catastrophic overfitting coincides with loss of local linearity, measured as gradient cosine similarity collapsing from about 0.95 to 0.05. Neither method touches Bellman targets, and neither treats the bound as a pessimism term.

Core Claim

Let $Q(s,a)$ be the critic, $g=\nabla_s Q(s,a)$ its state gradient, and $\kappa(s,a)$ an estimate of its largest state-curvature, meaning the top eigenvalue of $\nabla_s^2 Q$. A second-order expansion bounds the worst value within an $\ell_\infty$ ball of radius $\epsilon$ from below:

$$Q_{\epsilon}^{\downarrow}(s,a) \;=\; Q(s,a) \;-\; \epsilon\,\|g\|_1 \;-\; \tfrac{d\,\epsilon^2}{2}\,\max(\kappa,0),$$

Here $d$ is the state dimension, so $d\epsilon^2$ is the squared $\ell_2$ radius of the ball. We use $Q_{\epsilon}^{\downarrow}$ as a lower confidence bound of the form $\hat Q - \beta\Gamma$. The penalty $\Gamma$ is the perturbation sensitivity, and its scale is fixed by the threat radius $\epsilon$ rather than tuned as a free coefficient $\beta$. It enters in two places. The Bellman target becomes $r+\gamma\,Q^{\downarrow}_{\epsilon,\text{targ}}(s',\pi(s'))$. The actor maximizes $Q_{\epsilon}^{\downarrow}(s,\pi(s))$.

In the target, gradients are not propagated. $g$ therefore costs a single extra backward pass through the target network, with no double backpropagation. $\kappa$ is estimated by a finite difference along $g/\|g\|$, which costs one extra forward and one extra backward pass. Only the actor term needs a mixed second derivative, and it does so through a small MLP.

The bound is asymmetric by construction, because it only lowers values. This matches RORL's insight that smoothing must never reintroduce over-optimism, but without a separate asymmetric loss. States where the value is steep are backed up as worse, so the fixed point pushes the policy toward regions where the value is flat.

Research Program

Host algorithms. We implement the bound in two places.

  1. TD3+BC, a single twin-critic learner. Its attack results and wall-clock numbers are published in Towards Robust Policy.
  2. RORL, where the bound replaces the adversarial perturbation sampling while the 10-network ensemble is kept.

The second host answers a narrower question: can the expensive part of RORL be swapped out without losing its robustness?

Tightness checks. During training we log two diagnostics.

  • The gap between $Q_{\epsilon}^{\downarrow}$ and the true worst-case value, found offline with 50-step projected gradient descent (PGD). This measures bound tightness as $\epsilon$ grows, which is QUB's known weak point.
  • A GradAlign-style cosine between the state gradient at $s$ and at a random point in the ball. This detects loss of linearity in value networks.

Pessimism versus smoothness. A separate analysis addresses the tension that RORL points out between how pessimistic a method is and how smooth its value function stays. For CQL, EDAC-style ensembles, and our method at matched clean score, we measure the state-gradient norm and the finite-difference curvature of the learned critic. This gives the first direct measurement of how the choice of uncertainty penalty shapes value-function geometry.

Baselines

The do-nothing reference is plain TD3+BC trained on the same D4RL data. It fixes the clean score and the 3.05-hour training cost against which all overhead is reported. Towards Robust Policy reports that it loses up to 75% under the Robust Critic Attack.

The strongest competitor on robustness is RORL. Its clean D4RL Gym average is 85.7, against 82.9 for EDAC and 70.2 for CQL. Per-$\epsilon$ attacked scores appear in the file only qualitatively, so there is no reported numeric result found on Lacuna for RORL under attack, and we will rerun it ourselves.

The strongest competitor on the same host algorithm is the actor and critic defense of Towards Robust Policy, reported at +11% and +10% under attack with 4.55 hours of training.

The cheap smoothing competitor is the one-sided gradient penalty of TD3BC++. We run it as published, on action gradients, and also in a version extended to state gradients.

Fairness conditions.

  • All methods use the same datasets, network sizes, and one million gradient steps.
  • All methods run on the same GPU type.
  • Each method gets the same 20-configuration tuning budget, with its robustness radius fixed to the evaluation $\epsilon$.
  • Wall-clock time is always reported alongside scores.

Win conditions.

  • Against RORL: attacked scores at least as high, with no worse clean score, at clearly lower training time.
  • On TD3+BC: exceed the actor defense's attacked score while staying under roughly 1.5× baseline time. That is well below the 1.49× the defense itself costs only if we also beat it on score, and we will say so plainly if we do not.

Ablations isolating the key ingredient.

  1. Remove the curvature term, leaving a first-order bound only.
  2. Move the bound out of the target and actor objectives and into a plain regularizer on the critic.
  3. Replace the bound with one-step FGSM-perturbed evaluation.

Why not just…?

Why not just add a state-gradient norm penalty to the critic, like TD3BC++ but on states? This is the most serious threat, because it is cheap and the file contains no evidence against it. Our claim is that where the term enters matters. A regularizer flattens the critic everywhere, including where steepness is real, and it is known to trade off clean accuracy (the over-smoothing issue that QUB needed decreasing schedules for). Using the bound as pessimism only lowers backed-up values, and only by an amount set by $\epsilon$. Ablation 2 tests exactly this at matched overhead. If the regularizer matches our method, the honest conclusion is that the Bellman placement adds nothing, and we will report the regularizer as the recommended method.

Why not just use one-step FGSM perturbations in RORL's smoothing loss? It is the natural way to make RORL cheap. However, GradAlign and Understanding Catastrophic Overfitting show that single-step training in classifiers collapses to near-zero multi-step robustness once linearity breaks. We predict the same for value networks at larger $\epsilon$. The linearity cosine and the gap between one-step and PGD attacks, tracked during training, will show whether this collapse occurs. If it does not occur in offline RL, FGSM-RORL becomes a strong cheap baseline and our curvature term has to justify itself on tightness alone.

Why not just add Gaussian noise to states? Towards Robust Policy finds random perturbations far weaker than critic-guided ones as attacks. GradAlign also shows that random initialization does not prevent geometric breakdown. We include noise augmentation at matched variance as a baseline and expect it to fail under the Robust Critic Attack.

Why not just use more pessimism, such as CQL or a bigger ensemble? RORL shows point-wise conservatism worsens spikiness. EDAC's up to 50 networks cost memory without addressing state attacks. Our pessimism-versus-smoothness analysis measures this directly.

Proposed Experiments

Setup. We evaluate on D4RL Gym-MuJoCo (D4RL): Hopper, HalfCheetah, and Walker2d, each with medium, medium-replay, medium-expert, and expert data. The metric is normalized score over 5 seeds. Attacks are Random, Critic, Robust Critic, and Actor attacks from Towards Robust Policy, plus 50-step PGD against the learned critic. Each is applied at several $\epsilon$ values scaled by per-dimension state standard deviation, as in RORL. We report clean score, attacked score, and training hours. As a stress test of whether the bound helps in higher dimensions, we add Adroit's 24-DoF hand tasks.

Decisive checks. There are three.

  1. Our TD3+BC variant must close most of the attacked-score gap to RORL while costing close to 1× baseline time rather than 1.5× or more.
  2. The one-step FGSM variant must show a measurable catastrophic-overfitting signature that our method avoids. The signature is a falling linearity cosine and a widening gap between one-step and PGD attacks.
  3. The target-placement version must beat the plain state-gradient regularizer at equal cost.

If the third check fails, the proposal reduces to recommending the cheap regularizer. We regard that as a useful result and will publish it as such.