Robot controllers trained in simulation are often declared "robust to the reality gap" after surviving perturbations: nudges to their state, or shifts in friction and mass. Those perturbations are almost always produced by weak, fixed-step attacks, often a single gradient step. Image classification went through the same phase. Croce and Hein showed that fixed-step projected gradient attacks stall. When they replaced them with an ensemble of self-tuning attacks, reported robustness fell for 48 of 49 defended models, and fell by more than 30% in 8 cases (Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks). We propose to bring that lesson to robotics. The plan is to build a budget-aware attack suite that searches over physics and observation perturbations without hand-tuned hyperparameters. We would then use it to compute, for each policy, the smallest physically plausible perturbation that makes it fail.
The payoff is not only stricter bookkeeping. If a stronger attack finds a policy's true failure margin, that margin might be a cheap, simulation-only predictor of how much the policy will lose when deployed on real hardware. That drop is currently 24–30% on standard manipulation setups (Robot Policy Evaluation for Sim-to-Real Transfer: A Benchmarking Perspective), and nobody can forecast it before running the robot. Our two-sentence pitch is as follows. Robustness claims for sim-trained policies rest on under-tuned attacks, and a parameter-free attack ensemble will expose this the way AutoAttack did for classifiers. The minimum failure margin this ensemble finds should rank policies by real-world transfer loss better than average returns under randomized simulation do.
Core Claim
We make two falsifiable claims. The first is that policies trained by state-adversarial or parameter-adversarial robust RL look robust partly because they were evaluated with fixed-step perturbations. Those methods are RAPPO (Revisiting Domain Randomization via Relaxed State-Adversarial Policy Optimization), which perturbs states with a single value-gradient step, and Adv-USR (Robust Reinforcement Learning in Continuous Control Tasks with Uncertainty Set Regularization), which builds its physics uncertainty set from first-order gradients with heuristic bounds. An adaptive attack ensemble at the same rollout budget should find substantially lower worst-case returns for these policies.
The second claim is that the attack-found minimum failure margin tracks transfer loss. Define it as
$$\rho(\pi)=\min_{\delta}\ \|\delta\|_{W}\quad\text{subject to}\quad \mathrm{Success}(\pi,\ \theta_0+\delta)=0,$$
where $\pi$ is the policy, $\theta_0$ the nominal simulator parameters, and $\delta$ a perturbation that may vary over time. The perturbation covers friction, mass, damping, actuation latency, and observation offsets. $\|\cdot\|_W$ is a norm that rescales each parameter by its physically plausible range. Our claim is that $\rho(\pi)$ correlates more strongly, in rank, with each policy's sim-to-real performance drop than nominal return, mean randomized return, or margins found by fixed-step attacks.
The closest methodological ancestor is HO-FMN (Improving Fast Minimum-Norm Attacks with Hyperparameter Optimization). It tuned minimum-norm image attacks with automated hyperparameter search and obtained a continuous robustness-versus-perturbation curve. We borrow the minimum-norm framing but must drop its reliance on smooth gradients, because contact dynamics are not differentiable in a useful way.
Research Program
The attack suite has three members, chosen for complementary failure modes. The first is a gradient-based attacker that uses finite differences or a differentiable simulator. It follows the step schedule of APGD: an exploration phase, then a step size that halves automatically whenever progress stalls. This removes the step-size choice that makes single-step attacks brittle. The second is a gradient-free evolutionary attacker, playing the role that Square Attack plays in AutoAttack. It is the insurance against contact-induced gradient masking, and it borrows the population-and-mutation machinery that EASI (EASI: Evolutionary Adversarial Simulator Identification for Sim-to-Real Transfer) uses for system identification. We redirect it from matching reality to breaking the policy. The third is a minimum-norm boundary search that shrinks $\delta$ along the failure boundary once any attacker finds a failure, which gives $\rho(\pi)$.
The analogue of the loss-function problem deserves emphasis. Croce and Hein replaced cross-entropy with a scale-invariant logit-ratio loss because cross-entropy gradients vanish on confident predictions. Episode return has a similar flaw as an attack objective: it saturates when the policy is comfortably succeeding and gives no signal about how close failure is. We will use a normalized, per-step distance-to-failure margin instead. Examples are the minimum over time of torso height above the fall threshold, or grasp-slip distance relative to its nominal value. We will ablate this margin against raw return.
Perturbations are allowed to vary over time, for example latency spikes or friction that changes mid-contact. This is what makes the search space high-dimensional, and it is where we expect uniform sampling to be weakest.
Validation needs a stand-in for reality that the attacker cannot see. Otherwise the correlation claim becomes circular. We will build a proxy-reality protocol. Policies are trained and attacked in MuJoCo, Isaac, or ManiSkill2 with the standard parameterization. The "real" environment adds effects outside the attack's parameter space: stick-slip friction models, gear backlash, sensor drift, and correlated observation noise. These are the unmodeled phenomena catalogued in the embodied-manipulation literature (A Survey of Embodied Learning for Object-centric Robotic Manipulation). Where hardware is available, we add true physical transfer on the ManiSkill2 PickCube task. Its authors report 91% success in simulation and 60% on the physical robot (ManiSkill2: A Unified Benchmark for Generalizable Manipulation Skills). The policy population will be deliberately diverse: uniform and active domain randomization (Active Domain Randomization) with varied ranges, RAPPO, Adv-USR, and SPiDR-style variance-penalized policies (SPiDR: A Simple Approach for Zero-Shot Safety in Sim-to-Real Transfer) with several penalty weights, all across seeds. This yields roughly forty or more policies per task, enough for rank statistics.
Baselines
The naive reference point is nominal simulation return: evaluating the policy only at $\theta_0$. Any useful predictor of transfer loss must beat it.
The strongest practical competitor is randomized evaluation. Each policy is evaluated under uniform domain randomization over the same parameter ranges the attack may use. We report both mean return and the 5th-percentile return, at exactly the rollout budget the attack suite consumes. Wide uniform randomization is what practitioners actually do, and in low-dimensional parameter spaces random sampling is a strong search method.
The strongest existing adversarial evaluators are two. One is the single-step value-gradient state attack used by RAPPO. The other is the first-order parameter attack that constructs Adv-USR's uncertainty set. We run both at the settings described in their papers, and again with the same rollout budget as our suite, so that any gain cannot be attributed to extra compute. The file reports Adv-USR only as achieving the highest average rank on Robust-AUC across the RWRL suite, and reports RAPPO only as having the best average rank on MuJoCo. No reported numeric robust-return value was found on Lacuna for either method, so we will reproduce both and report absolute numbers ourselves.
A stronger off-the-shelf black-box optimizer, CMA-ES or Bayesian optimization over the static parameters with default settings, serves as a further baseline.
Three ablations isolate the key ingredients. The first removes adaptive step halving. The second replaces the normalized margin loss with raw return. The third restricts perturbations to static rather than time-varying ones.
Beating the baselines means two things. For evaluation, it means lower worst-case success at equal budget on a majority of policy families. For prediction, it means a Spearman correlation between $\rho(\pi)$ and proxy-reality drop that exceeds the best baseline predictor by a margin outside the bootstrap confidence interval.
Why not just…?
Why not just widen domain randomization and report the worst sampled outcome? This is the most serious threat. With a handful of static parameters, it may genuinely match the attack. Evidence against uniform randomization as a training method is strong: it scored 0% where auto-tuned search reached 90% and 40% on xArm tasks (Auto-Tuned Sim-to-Real Transfer), and it produced 26.1x more safety violations than SPiDR on the Go1. That evidence concerns training, however, not evaluation. Our budget-matched comparison settles the evaluation question directly. If uniform sampling finds failures as often as the attack in the static setting, we will say so. We would then rest the evaluation claim on time-varying perturbations, where the search space grows with episode length and random sampling should degrade.
Why not just use a standard black-box optimizer as the attacker? It is included as a baseline and as a component of the ensemble. The contribution is not any single optimizer. It is the combination of step-size adaptation, a non-saturating margin objective, minimum-norm boundary search, and validation against held-out reality. The loss and schedule ablations show whether those pieces matter beyond picking CMA-ES.
Why not just identify the real parameters with EASI or DEAL and evaluate there? Those methods, EASI and DEAL: Diffusion Evolution Adversarial Learning for Sim-to-Real Transfer, find the nominal parameters that make the simulator match reality. That requires real rollouts, and it says nothing about the region just beyond the identified point, where the unmodeled effects live. Our proxy reality contains effects no parameter setting can reproduce. If system-identified evaluation predicted the drop as well as $\rho(\pi)$, the proxy experiments would show it, and we would report it as a win for identification.
Why not just measure transfer on hardware? Hardware transfer is the ground truth we calibrate against. It costs orders of magnitude more per policy, however, and cannot be used to screen dozens of candidates or to select penalty weights such as SPiDR's $\lambda$, which its authors tune by hand.
Proposed Experiments
The first experiment is a re-evaluation study. We re-evaluate RAPPO on the five MuJoCo locomotion tasks and Adv-USR on the RWRL suite with the full attack ensemble at matched budgets, and report the fall in worst-case return relative to each paper's own attack. The decisive check is the AutoAttack result reproduced in miniature: a substantial fraction of "robust" policies losing large amounts of worst-case performance. A null result, with the original attacks already near-optimal, would itself be informative, and we would publish it.
The second experiment is a prediction study. It runs on the proxy-reality versions of Hopper, Walker2d, the RWRL tasks, and ManiSkill2 PickCube. We compute $\rho(\pi)$ and every baseline predictor for each policy, then report Spearman correlation with transfer drop, with bootstrap intervals. We pre-register that the claim fails if $\rho(\pi)$ does not beat the 5th-percentile randomized return. On PickCube, a smaller subset of policies is transferred to the physical arm to check that proxy-reality rankings survive contact with an actual robot.
The third experiment is a use-case test. We select SPiDR's penalty weight and RAPPO's relaxation weight by maximizing $\rho(\pi)$ subject to a nominal-return floor, rather than by trial and error. We then test whether the chosen settings transfer better than the published defaults.
References
- Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks
- Robot Policy Evaluation for Sim-to-Real Transfer: A Benchmarking Perspective
- Revisiting Domain Randomization via Relaxed State-Adversarial Policy Optimization
- Robust Reinforcement Learning in Continuous Control Tasks with Uncertainty Set Regularization
- Improving Fast Minimum-Norm Attacks with Hyperparameter Optimization
- EASI: Evolutionary Adversarial Simulator Identification for Sim-to-Real Transfer
- A Survey of Embodied Learning for Object-centric Robotic Manipulation
- ManiSkill2: A Unified Benchmark for Generalizable Manipulation Skills
- Active Domain Randomization
- SPiDR: A Simple Approach for Zero-Shot Safety in Sim-to-Real Transfer
- Auto-Tuned Sim-to-Real Transfer
- DEAL: Diffusion Evolution Adversarial Learning for Sim-to-Real Transfer