Robots are increasingly asked to do missions with order and deadlines. A typical one reads: "reach the charging zone, then the inspection zone within 20 seconds, and never enter the restricted area." Signal Temporal Logic (STL), a formal language for such time-bounded requirements, scores a trajectory with a single number called robustness. The score is positive by how much the trajectory satisfies the mission and negative by how much it violates it. Smoothed versions of this score can be differentiated, so a policy can be trained directly against the mission rather than against a hand-written reward. Separately, domain randomization trains a policy in simulation across many sampled physical parameters (friction, mass, actuator delay) so it works on real hardware without any real-world training. The two have not been put together in the obvious way. Domain randomization averages reward over sampled physics, and averaging is the wrong operation for a mission that must hold on the real robot, whose physics is one unknown draw.
We propose to train closed-loop policies whose objective is the worst-case robustness over a set of physics parameters. Gradients do two jobs. They update the policy, and they also search the physics parameters themselves for configurations that break the mission. Those "counterexample physics" are added to a growing training set. The claim is concrete: a policy trained this way on a 1/10-scale race car and a simulated quadrotor will satisfy timed multi-stage missions on held-out and real physics more often than uniform randomization, risk-sensitive randomization, or per-step safety pessimism. It should also need far fewer sampled environments, and it should not become sluggishly overcautious.
What exists and what is missing
The closest work is Robust Counterexample-guided Optimization for Planning from Differentiable Temporal Logic. It frames planning as a game. A planner optimizes a trajectory to satisfy smoothed STL, while an adversary backpropagates through a differentiable simulator to find additive disturbances (wind-like forces) that lower robustness, and these accumulate as counterexamples. On satellite rendezvous it needed a median of 2 counterexamples where uniform randomization needed over 64 samples, and it kept positive robustness in over 90% of adversarial trials. That result motivates this proposal. Its limits are what we change. It produces an open-loop plan, not a feedback policy, and its adversary perturbs additive forces, not the structural parameters (friction, mass, latency) that actually make up the reality gap. A feedback policy behaves differently under wrong physics than a plan does, because it can react. An adversary over physics parameters is therefore searching a different and more relevant space.
Two lines supply the method. Backpropagation through signal temporal logic specifications turns STL formulas into differentiable computation graphs (stlcg) that batch on GPUs. We use this to get gradients of robustness with respect to both policy weights and physics parameters. Accelerated Learning with Linear Temporal Logic using Differentiable Simulation shows that analytic policy gradients through a differentiable simulator with softened logic reach good CartPole policies in 20M steps, while PPO and SAC learn nothing useful in 100M. It also reports doubled returns on Cheetah and Ant. However, it trains on one fixed simulator and never transfers to hardware, and that is exactly the gap we fill.
Two further works motivate the problem from the randomization side. SPiDR shows that plain randomization is risk-neutral near safety boundaries: it produces 6.2x more violations on a RaceCar and 26.1x more on a Unitree Go1. SPiDR fixes this by penalizing per-step costs with ensemble disagreement and reaches 100% real-world constraint satisfaction. Its constraints are single-step, though, with no ordering or deadlines. Active Domain Randomization steers parameter sampling toward environments where behavior diverges from a reference, using a learned discriminator. That signal does not know whether the divergence helps or breaks the task.
One more line constrains the design. Distributionally Robust Control for Chance-Constrained STL proves that exact STL robustness is Lipschitz, but its guarantees stop at linear systems. We do not claim formal guarantees. Instead, we evaluate everything with exact, unsmoothed robustness, because smoothed scores can be positive on trajectories that actually violate the mission.
Core Claim
Let $\pi_\theta$ be a feedback policy with weights $\theta$. Let $\xi \in \Xi$ be a vector of physical parameters inside a box $\Xi$, the same box ordinary randomization would sample from. Let $\tilde\rho(\theta,\xi)$ be the smoothed STL robustness of the closed-loop rollout under physics $\xi$. We train on
$$ \max_\theta \; \Big[(1-\beta)\,\mathbb{E}_{\xi\sim U(\Xi)}\,\tilde\rho(\theta,\xi) \;+\; \beta \min_{\xi\in\mathcal{C}}\tilde\rho(\theta,\xi)\Big], $$
where $\mathcal{C}$ is the growing set of counterexample parameters and $\beta \in [0,1]$ trades average-case against worst-case performance.
New counterexamples come from projected gradient descent on $\xi$ to minimize $\tilde\rho(\theta,\xi)$, started from several random points in $\Xi$. A candidate is kept only if its exact robustness is negative. Uniform samples prevent the policy from overfitting to a handful of adversarial worlds. The counterexample term forces satisfaction at the edges of the box, where averaging gives almost no training signal.
The claim is that gradient search over physics parameters finds mission-breaking corners of $\Xi$ far more efficiently than sampling does. This should hold most strongly when the failure set is a small region of a parameter space with 6 to 10 dimensions, and that is the regime real robots live in.
Research Program
We build the system in MuJoCo MJX, the differentiable JAX simulator SPiDR already uses, so that its RaceCar setup can be reproduced. Randomized parameters include tire friction, mass, motor gain, steering offset, and actuation delay. Delay is made differentiable with fractional-step interpolation between buffered actions. Missions are written in STL and compiled through an stlcg-style graph with log-sum-exp smoothing.
The policy can be updated two ways. The first is analytic gradients through short-horizon rollouts, as in Accelerated Learning with Linear Temporal Logic using Differentiable Simulation. The second is PPO with exact robustness as a terminal reward, where gradients are used only for the adversary over $\xi$. Running both separates two questions: does the adversary help, and do analytic policy gradients help?
The analysis has three parts. First, we measure how often gradient-found counterexamples are real, meaning exact robustness is negative, versus artifacts of smoothing; we sweep the smoothing sharpness $k$. Second, we measure how the size of $\mathcal{C}$ grows over training. Third, we measure where in $\Xi$ counterexamples cluster, and whether those regions predict the parameters identified on the real car.
Contact-rich quadruped dynamics are a known weak point for simulator gradients. The Go1 is therefore a stretch target. The core claims rest on the car and on a simulated quadrotor.
Baselines
The naive reference is uniform domain randomization over the same box $\Xi$, trained with PPO. Its reward is the robustness score itself, a hand-shaped dense reward, or both, whichever does better under equal tuning. Following Sim-to-Real: Learning Agile Locomotion For Quadruped Robots, this is what most practitioners would actually deploy. We also train on nominal physics only, to show how large the reality gap is.
The strongest existing method for zero-shot safety is SPiDR. It is the strongest competitor because it already beats randomization on the same RaceCar hardware. It reports 100% real-world constraint satisfaction and 6.2x fewer violations than randomization. We run it with its safety cost set to the per-step parts of the mission (avoid-region distances), and with the ordering and deadline parts as reward. Its weight $\lambda$ gets the same tuning budget as our $\beta$.
We also run Active Domain Randomization with its discriminator-based sampler. A second ADR variant replaces the discriminator with negative exact robustness as the sampler reward. This variant isolates whether gradients matter, or whether a robustness-aware black-box search over physics is enough.
The key ablation is our method minus the counterexample term ($\beta=0$). That is plain uniform randomization with a differentiable STL objective, and the comparison isolates the adversary.
All methods share the simulator, parameter box, policy architecture, and number of simulated environment steps, with matched wall-clock compute. Each gets 20 hyperparameter configurations. Each is evaluated on the same held-out set of 2,000 parameter samples. That set deliberately includes 10% of samples drawn from slightly outside $\Xi$, to probe physics the training box does not cover.
Beating SPiDR means two things. On hardware, it means equal or higher full-mission satisfaction, deadlines included, not just obstacle avoidance. In simulation, it means a higher 5th-percentile exact robustness, with mission completion time no worse than SPiDR's.
Why not just…?
Why not just use uniform randomization with a risk-sensitive objective, such as the average of the worst 10% of sampled robustness scores? This is the most serious threat. Simulators on a GPU make huge batches cheap, and a tail average needs no adversary. Our bet is that when the failure set is a thin slice of a parameter box with 6 to 10 dimensions, even large batches rarely land in it. Dawson et al.'s comparison, 2 counterexamples against over 64 samples with lower success for sampling, points the same way, but it was measured on disturbances rather than physics. We include this tail-average baseline at batch sizes up to 4,096 environments. If it matches our held-out tail robustness at equal compute, the adversary is not needed. In that case the contribution shrinks to "train on a tail of exact robustness," and we will report that honestly.
Why not just add SPiDR-style pessimism to the mission's safety terms? SPiDR penalizes each step independently. A deadline like "reach B within 20 s after A" cannot be split into per-step costs without inventing a reward, which brings back the hand-shaping problem. Per-step pessimism also tends to slow the robot down, and slowing down is exactly what makes deadlines fail. The hardware comparison will show whether SPiDR keeps obstacle clearance but misses deadlines, which is the failure we predict.
Why not just plan open-loop with counterexample-guided STL and replan often? Robust Counterexample-guided Optimization for Planning from Differentiable Temporal Logic reports solutions in under 4 minutes, which is too slow to replan at control rate on a car at speed. Its adversary also covers additive forces, not delay or friction. We include it as a baseline that replans at whatever rate it can sustain, to see what feedback adds.
Why not just use mixed-integer programming for exact STL? The same paper reports mixed-integer solvers timing out after an hour on satellite missions. They also need linearized dynamics, so they do not fit a policy over nonlinear simulated physics.
Proposed Experiments
In simulation, we test a mission on the RaceCar model: visit two gates in order within deadlines while always avoiding a region. We also test the quadrotor and 29-dimensional Ant missions from the suite used by TGPO, with randomized mass, drag, and delay. TGPO reports over 61% success on the 29D Ant without randomization, which serves as a reference for how hard the nominal task is. The metrics are exact satisfaction rate, 5th-percentile exact robustness on held-out physics, mission completion time as a measure of overcaution, and simulated steps to reach a target tail robustness.
On hardware, we run 30 trials per method on the 1/10-scale car on a taped course, logging full-mission satisfaction from motion capture. The no-adversary ablation is also run on the real car. The first decisive check is whether our method beats the large-batch tail-average baseline on held-out tail robustness at equal compute. The second is whether it beats SPiDR on real-car full-mission satisfaction without slower completion. If both fail, gradient search over physics adds nothing over sampling, and the proposal is wrong.
References
- Robust Counterexample-guided Optimization for Planning from Differentiable Temporal Logic
- Backpropagation through signal temporal logic specifications: Infusing logical structure into gradient-based methods
- Accelerated Learning with Linear Temporal Logic using Differentiable Simulation
- SPiDR: A Simple Approach for Zero-Shot Safety in Sim-to-Real Transfer
- Active Domain Randomization
- Distributionally Robust Control for Chance-Constrained Signal Temporal Logic Specifications
- Sim-to-Real: Learning Agile Locomotion For Quadruped Robots
- TGPO: Temporal Grounded Policy Optimization for Signal Temporal Logic Tasks