Legged robots run best when a neural network sends torques straight to the motors, with no position-tracking loop in between. Those policies absorb kicks and cross rough ground better than position-controlled ones. They are also the policies most likely to damage hardware. A small mismatch in friction, latency, or motor voltage can make them chatter, slam a joint into its stop, or spike current. Domain randomization trains a policy across many simulated physics settings so that it transfers to a real robot without fine-tuning. Even when combined with safety constraints, it usually bounds the cost averaged over those settings, so a policy can be safe in most simulated worlds and violent in a few. We think the real robot often sits in exactly that region of bad actuator parameters.
We propose to train torque-level locomotion policies under a constraint on the tail of the distribution of simulated worlds. The expected safety cost in the worst $\alpha$ fraction of randomized physics settings must stay below a budget. A sampler deliberately concentrates training rollouts on the settings where the policy currently violates its limits. All enforcement happens at training time, so the deployed 200 Hz controller carries no runtime safety filter. The claim is that, for torque control, constraint violations concentrate in a narrow band of actuator parameters. If that holds, constraining the tail buys real-world safety at a small cost in agility. Penalties that act on every state, such as uniform pessimism or reward shaping, pay that cost everywhere.
Where this sits
Two results motivate the work. Direct torque policies trained with plain PPO and friction/push randomization transfer zero-shot on a Unitree A1 at 200 Hz and recover from kicks that topple position controllers (Learning Torque Control for Quadrupedal Locomotion). On the 100 kg TOCABI biped, torque policies cleared 1 cm obstacles with 100% success where position control failed 40% of the time (Torque-based Deep Reinforcement Learning for Task-and-Robot Agnostic Learning on Bipedal Robots Using Sim-to-Real Transfer). Neither paper states or measures a safety constraint. Both rely on unconstrained optimization with action scaling or gravity-compensation pre-training to keep early exploration from producing torque spikes.
The closest prior work is SPiDR: A Simple Approach for Zero-Shot Safety in Sim-to-Real Transfer. It couples randomization with a constrained objective by adding a penalty proportional to how much an ensemble of randomized simulators disagree about each transition. It reports 100% real-world constraint satisfaction on a Go1 quadruped and 26.1x fewer violations than plain randomization. Our proposal differs from SPiDR in two ways. First, SPiDR ran the Go1 through a position-setpoint interface, which is the abstraction that already filters out most actuator-level hazards. Second, SPiDR penalizes disagreement wherever it occurs, including places where every simulated world disagrees but all of them are safe, and its authors acknowledge this can produce a frozen "safe but useless" agent. A tail constraint only penalizes behavior in worlds that actually incur cost. That is the sharpest testable difference between the two.
Two other lines of work constrain the design. Runtime shields, such as Hamilton–Jacobi reachability backups in Sim-to-Lab-to-Real: Safe RL with Shielding and Generalization Guarantees and the viability filter of Viability-Preserving Passive Torque Control, work at the navigation level or assume nominal dynamics. The latter's 155 Hz loop on a fixed arm, against 66 Hz for a barrier-function baseline, shows how tight the compute budget gets at torque rates. We therefore move safety into training. Provably Safe Sim-to-Real Transfer gives guarantees only for tabular problems, but it supplies a useful idea: real-world risk depends on the size of the mismatch region, not on the whole state space. Our concentration hypothesis is the continuous, empirical analogue of that idea.
Core Claim
Let $\xi$ denote a vector of simulator parameters (joint friction, motor torque constant, bus-voltage limit, sensing and actuation latency, payload mass, ground friction) drawn from the randomization distribution $p(\xi)$. Let $C(\pi,\xi)$ be the expected discounted safety cost of policy $\pi$ in world $\xi$. The cost function marks four kinds of event: joint positions inside a margin of their hard stops, time at torque saturation, torque rate above a chatter threshold, and estimated winding temperature from a first-order heating model over a limit. Body or knee contact with the ground also counts. Standard constrained randomization bounds the average $\mathbb{E}_{\xi}[C(\pi,\xi)]$. We instead require
$$ \mathrm{CVaR}_\alpha\big[C(\pi,\xi)\big] \;=\; \mathbb{E}_{\xi}\big[\,C(\pi,\xi)\;\big|\;C(\pi,\xi)\ge q_\alpha(\pi)\,\big] \;\le\; \epsilon . $$
Here $q_\alpha(\pi)$ is the cost level exceeded by only the worst $\alpha$ fraction of worlds, and $\epsilon$ is the safety budget. Setting $\alpha=1$ recovers the mean constraint. As $\alpha$ approaches 0 the constraint approaches worst-case robustness, which is known to make policies sluggish when randomization ranges are wide. The prediction is that intermediate $\alpha$ (we will sweep 0.05–0.3) gives near-mean-constraint agility with near-worst-case safety. The reason is specific to torque control: without a PD loop acting as a low-pass filter, instability appears only when latency and friction push the policy's effective feedback gain past a margin. That is a small subset of parameter space.
Research Program
The first deliverable is an empirical map rather than an algorithm. We train the unconstrained torque policy recipe from the A1 paper with widened actuator randomization. We then evaluate it on a dense grid of held-out $\xi$ and record where each cost type is incurred. If violations turn out to be spread evenly across parameter space, the tail constraint has no advantage over a mean constraint, and we will report that as the main finding.
The second deliverable is the training method. Each PPO iteration samples $N$ worlds, where $N$ is the number of parallel simulated environments in Isaac Gym. We estimate per-world episode costs, select the worst $\alpha N$, and update a Lagrange multiplier on their mean cost. Policy updates use a first-order constrained step in the style of First Order Constrained Optimization in Policy Space, which avoids the oscillation that Safety Gymnasium: A Unified Safe Reinforcement Learning Benchmark documents for plain Lagrangian methods.
Because the tail is by definition rare, a naive estimate is noisy. We therefore fit a lightweight classifier over $\xi$ that predicts violation probability. A fraction of worlds is drawn from regions the classifier rates as near the decision boundary, and importance weights keep the tail estimate unbiased with respect to $p(\xi)$. This sampler is also our answer to the "safety paradox" raised in Breaking Safety Paradox with Feasible Dual Policy Iteration: safe learners starve their cost estimators of failures. In simulation, failures are free, so we seek them out deliberately. "Safe exploration" here means that the robot's first real rollouts respect the constraints, not that simulated training does.
The third deliverable is a zero-shot hardware evaluation on the A1. It includes rollouts with deliberately degraded actuators, so that the real robot is pushed toward the tail rather than tested only near nominal conditions.
Baselines
The naive reference point is the unconstrained torque recipe of Learning Torque Control for Quadrupedal Locomotion. It uses PPO, 200 Hz control, and 10x action scaling, and we add hardware torque clipping. We run it with exactly our randomization ranges, network, and simulation step count. That paper reports kick recovery and outdoor transfer but no violation counts, so there is no reported result on Lacuna for its safety cost. We will measure that cost ourselves. Beating this baseline means fewer real-world violations per minute at no more than a 10% loss in commanded-velocity tracking.
The second baseline is the same policy with large reward penalties on each cost term, tuned over the same budget of hyperparameter trials as our $\alpha$ and $\epsilon$. It represents the shaping practice used in most locomotion work.
The third is mean-constrained randomization with PPO-Lagrangian from SafePO. On velocity tasks, the Safety Gymnasium authors report a 98% reduction in violations with a 45% reward drop relative to PPO. That figure is the reference for how costly a mean constraint can be, although it was measured on different tasks.
The strongest competitor is SPiDR reimplemented on the torque action space. It uses the same ensemble of randomized simulators, with its disagreement penalty weight tuned under the same trial budget we give our own method. The reported number to beat comes from its Go1 position-level experiments: 100% constraint satisfaction and 26.1x fewer violations than plain randomization. We have found no reported result on Lacuna for SPiDR on a torque-level policy. Beating it means matching its real-world violation rate with higher velocity-tracking reward and a lower freezing rate, defined as the fraction of commands the robot refuses to follow.
A position-controlled policy trained with our same tail constraint isolates whether the torque action space pays off once safety is enforced. Two ablations isolate the key ingredients. The first removes the adaptive sampler, keeping the tail constraint with uniform world sampling. The second keeps the sampler but uses a mean constraint. All methods get the same simulation steps and seeds (five per method).
Why not just…?
Why not just clip and rate-limit torques at the motor driver? Clipping removes current spikes, but it does nothing about joint-limit impacts, falls, or heat building up under sustained near-saturation. Clipping also changes the closed-loop dynamics the policy was trained on, which can itself cause oscillation. We include clipping in every method, so any remaining violations are by construction ones clipping cannot fix. The violation map in the first deliverable shows how much of the cost is torque saturation versus other causes. If saturation dominates, this alternative wins, and we will say so.
Why not just return to position control, which is inherently gentler? The TOCABI results, a 40% failure rate for position control on 1 cm obstacles, and the A1 kick tests argue that position loops give up robustness. But neither study held safety constraints equal across the two interfaces. Our tail-constrained position baseline does exactly that. If it matches the torque policy on both cost and disturbance recovery, the case for torque-level control under safety requirements weakens. That is a real possibility we will report.
Why not just apply SPiDR to torques? This is the most serious threat. Disagreement penalties may work just as well if torque-level violations happen to coincide with simulator disagreement. Our argument against it rests on the freezing failure SPiDR's authors acknowledge, and the head-to-head comparison above decides the question. The decisive comparison is the reward–violation frontier traced by sweeping SPiDR's penalty weight against sweeping our $\alpha$.
Why not just add a runtime shield? Reachability and barrier-function filters either assume nominal dynamics or cost loop rate, as the 66 Hz versus 155 Hz contrast in the viability paper shows. Braking-to-stop safe sets are also not well defined for a robot in stance. A trained-in constraint has zero inference overhead. A shield could still be layered on top later.
Decisive Checks
We will run four checks, ordered so that the cheapest can falsify the idea first. The concentration map comes first. If the worst 10% of worlds account for less than about a third of total cost, the core claim fails. Second, simulated tests on actuator parameters 25% outside the training band measure whether tail constraints degrade more gracefully than mean or disagreement constraints when the support assumption, shared with SPiDR, is broken. Third, a low-dimensional replication on safe-control-gym quadrotors with direct thrust commands and mass/inertia randomization makes the comparison reproducible without hardware. Fourth, zero-shot A1 deployment runs nominally and with an added 15 ms actuation delay, a lowered voltage limit, and a 3 kg payload. We will count joint-margin entries, saturation time, chatter episodes, thermal-model exceedances, and falls per minute, alongside velocity-tracking error. The proposal succeeds if, under the degraded conditions, our method keeps violations at or near zero while velocity tracking stays within 10% of the unconstrained torque policy, and SPiDR at the same violation rate tracks worse.
References
- Learning Torque Control for Quadrupedal Locomotion
- Torque-Based Deep Reinforcement Learning for Task-and-Robot Agnostic Learning on Bipedal Robots Using Sim-to-Real Transfer
- SPiDR: A Simple Approach for Zero-Shot Safety in Sim-to-Real Transfer
- Sim-to-Lab-to-Real: Safe RL with Shielding and Generalization Guarantees
- Viability-Preserving Passive Torque Control
- Provably Safe Sim-to-Real Transfer
- First Order Constrained Optimization in Policy Space
- Safety Gymnasium: A Unified Safe Reinforcement Learning Benchmark
- Breaking Safety Paradox with Feasible Dual Policy Iteration
- Safe-Control-Gym: A Unified Benchmark Suite for Safe Learning-Based Control and Reinforcement Learning in Robotics