A single neural network can now control many different robot bodies: quadrupeds from 2 kg to 200 kg, or a hundred procedurally generated walkers. Each new body still carries hidden human labor, though. The reward these controllers learn from is a hand-tuned sum of a dozen terms (forward speed, body tilt, foot clearance, torque, slip), and the right weights differ from body to body. Because nothing in training limits what the robot may do while learning, its safety on real hardware rests on whatever robustness happens to emerge. This proposal argues that most of those reward terms are constraints in disguise. They say "do not exceed this torque" or "do not tip over," not "more of this is better."
We propose to rewrite them as a few unit-free constraints, meaning quantities divided by a body-specific scale such as the actuator's torque limit or the robot's weight, so that one threshold makes sense for every body. A constrained learner then enforces these limits separately on each training body. To cover the gap between simulation and reality, we add a pessimism penalty that grows wherever randomized simulators disagree about what happens next. If this works, a universal controller would learn from a one-term reward: track the commanded velocity. It would satisfy physical limits on bodies it has never seen. It would do this without anyone retuning weights per robot.
Core Claim
The claim is specific and falsifiable. On a multi-morphology locomotion suite, compare two training setups. The first uses a velocity-tracking reward, five unit-free constraints with thresholds shared across all bodies, a separate Lagrange multiplier per (body, constraint) pair, and a disagreement penalty normalized per joint. A Lagrange multiplier is a learned penalty weight that rises while a constraint is violated and falls once it is satisfied. The second is the hand-tuned multi-term reward with domain randomization, as used by ManyQuadrupeds. We claim the first setup reaches comparable task performance with far fewer constraint violations, both on held-out bodies and on a physical quadruped. We also claim it needs no per-body hyperparameter search.
Written as a constrained problem, for every training morphology $m$ and constraint $k$:
$$\max_\pi \; \mathbb{E}_m \big[J_r(\pi; m)\big] \quad \text{s.t.} \quad J_{c_k}(\pi; m) \le d_k \;\; \forall m, k,$$
Here $J_r$ is the expected discounted velocity-tracking return, $J_{c_k}$ is the expected discounted value of cost $c_k$, and the threshold $d_k$ is shared across bodies because $c_k$ is unit-free. The pessimistic cost follows SPiDR:
$$\tilde c_k = c_k + \lambda\, \bar\upsilon,$$
Here $\bar\upsilon$ is the ensemble's next-state variance, divided per joint by that joint's range of motion. We expect this normalization to let a single $\lambda$ serve all bodies.
Positioning
The proposal builds on three cited works, uses two more for its method, and is shaped by two constraints from the literature.
Motivating work. ManyQuadrupeds trains one policy for 16 quadrupeds and transfers it zero-shot to the Unitree A1 and Go1. It uses a hand-tuned reward and imposes no safety constraints during training. GCNT uses a graph-transformer policy, which reads the robot as a graph of limbs and joints so it can handle any body layout. It controls varied bodies on the UNIMAL and SMPENV suites, again optimizing unconstrained return. We keep their architectures and replace their objective.
Closest prior method. SPiDR adds ensemble disagreement to the safety cost and reaches full constraint satisfaction on a real Go1. That work covers a single robot, keeps a hand-engineered task reward, and tunes $\lambda$ by hand. Its ensemble is tied to one fixed state and action space. We compute disagreement per joint on graph-structured states and test whether per-joint normalization removes per-body tuning of $\lambda$.
Method sources. The per-(body, constraint) multipliers use the PID-damped Lagrangian update that Safety Gymnasium shows reduces training oscillation.
Constraints from prior work. MetaDrive documents that fixed negative-reward penalties give either reckless or "safe but useless" policies. This is why we use adaptive multipliers rather than fixed penalties. Breaking Safety Paradox with Feasible Dual Policy Iteration warns that as a policy becomes safe, violating samples vanish and the learner can no longer locate the safety boundary. This matters more with many bodies, because each body's violation signal is sparse. We address it in the analysis below.
Contrast with other ways to avoid reward engineering. Adversarial pursuit-evasion training removes reward engineering through a game, but the resulting gaits are jittery and energy-inefficient. Hierarchical apprenticeship learning replaces reward tuning with human advice for a single robot. Neither imposes physical limits during learning. Our approach keeps a minimal task reward and moves the shaping into constraints.
Research Program
Constraints. We first convert the standard locomotion reward terms into five unit-free costs:
- peak joint torque divided by the actuator limit;
- distance to the joint-angle limit divided by joint range;
- body tilt;
- foot slip speed divided by leg length;
- peak contact force divided by body weight.
Each cost becomes a hinge: it is zero below a margin and grows above it. Any term that resists this rewrite is a genuine preference, not a limit. One example is gait style. Such terms stay in the reward, and we report how many survive.
Training. We train two policy families. The first is a GCNT-style graph transformer on UNIMAL, which has many body layouts and joint-level actions. The second uses the ManyQuadrupeds three-level hierarchy on its 16 quadrupeds, the route to hardware. Both are trained with Lagrangian PPO and multipliers maintained per (body, constraint) pair.
Disagreement ensemble. Eight or more physics-randomized simulator instances supply the disagreement signal, following SPiDR. Variance is computed per joint and normalized by joint range. For held-out bodies the policy is evaluated with no further training. The question is whether constraints learned on 100 bodies carry over to bodies the policy never saw.
Safety-paradox analysis. We track the fraction of rollouts near each constraint boundary over training, separately for each body. This tests whether violation data vanishes unevenly, starving some bodies of boundary information.
Baselines
Hand-tuned reward with domain randomization. This is the naive reference point and the current practice. It is the ManyQuadrupeds reward and hierarchy with its domain randomization, run on the same bodies with the same simulation steps. We give it the same number of training runs for its hyperparameter sweep as our method gets in total. Beating it means matching its velocity-tracking error within a pre-registered margin, set before running so we cannot pick the cutoff afterward, while showing far fewer constraint violations on held-out bodies. No reported scalar task-return number was found on Lacuna for ManyQuadrupeds. Its reported hardware result is that the A1 carried an extra 15 kg, a 125% increase over its own weight. We will rerun it ourselves.
Per-robot SPiDR specialists. This is the strongest existing safety method. It trains a separate SPiDR policy, with $\lambda$ tuned per robot, for each evaluation body, including the held-out ones. It therefore gets privileged access to every test body. SPiDR reports 100% real-world constraint satisfaction on the Go1, with standard domain randomization incurring 26.1x more violations. We treat specialists as an upper reference for safety, not a target to beat. Success means the universal policy comes within a small factor of the specialists' violation rate on held-out bodies at a fraction of their total compute and tuning.
Hand-tuned reward plus the same constraints. This ablation keeps the full multi-term reward and adds our constraints and pessimism. It isolates whether removing reward terms matters, or whether only the constraints do the work.
One shared multiplier per constraint. This ablation shares a single multiplier per constraint across all bodies, isolating the per-body dual variables.
Unnormalized disagreement. This ablation uses raw ensemble variance with $\lambda$ swept globally, isolating our normalization against SPiDR's original form.
For context on what constraint enforcement typically costs: Safety Gymnasium reports that PPO-Lag cut violations by 98% at a 45% loss in reward on velocity tasks. A reward loss far smaller than that would indicate the unit-free constraints are not over-restrictive.
Why not just…?
Why not just tune the penalty weights once and reuse them across bodies? That is what current universal controllers effectively do. MetaDrive and Safety Gymnasium both show that fixed penalties trade catastrophic violations against frozen behavior. SPiDR shows that domain randomization without constrained optimization violates far more on hardware. The first baseline measures this directly, with an equal tuning budget.
Why not just use one Lagrange multiplier per constraint? This is the real threat. Because our costs are unit-free, a shared multiplier might suffice, and it would be simpler. The shared-multiplier ablation settles the question. If its spread of per-body violation rates across the 100 UNIMAL bodies matches the per-body version, we will adopt the shared form and say so. We expect it to over-constrain light, stable bodies and under-constrain heavy, tall ones. This prediction can fail.
Why not just train SPiDR separately for each robot? Specialists cannot act on a body they were not trained for, and each one needs its own $\lambda$ search. ManyQuadrupeds and GCNT show shared policies are competitive with dedicated ones. The comparison asks what safety is lost by sharing, and we report it honestly even if the gap is large.
Proposed Experiments
UNIMAL. We train on 100 bodies across flat, incline, mixed-terrain, and obstacle environments, and evaluate zero-shot on held-out bodies. Metrics are velocity-tracking error and per-constraint violation rate, both reported per body. We also count hyperparameter configurations searched. On SMPENV we test variants with missing limbs, a structural change the per-body multipliers never saw.
ManyQuadrupeds hierarchy. We hold out its three extreme robots as the transfer test.
Hardware. We deploy to a Unitree Go1 and log torque, tilt, and slip against the same thresholds, with and without the 15 kg payload condition.
Decisive outcomes. The proposal fails if any of three results appear. First, the minimal-reward policy tracks velocity markedly worse than the hand-tuned baseline. Second, the constraints-on-full-reward ablation matches it, meaning removing reward terms buys nothing. Third, held-out violation rates are no better than domain randomization. It succeeds if held-out bodies and the real Go1 show violation rates near the per-robot SPiDR reference, with no per-body tuning.
References
- ManyQuadrupeds: Learning a Single Locomotion Policy for Diverse Quadruped Robots
- GCNT: Graph-Based Transformer Policies for Morphology-Agnostic Reinforcement Learning
- SPiDR: A Simple Approach for Zero-Shot Safety in Sim-to-Real Transfer
- Safety Gymnasium: A Unified Safe Reinforcement Learning Benchmark
- MetaDrive: Composing Diverse Driving Scenarios for Generalizable Reinforcement Learning
- Breaking Safety Paradox with Feasible Dual Policy Iteration
- Learning Agile Locomotion via Adversarial Training
- Hierarchical Apprenticeship Learning with Application to Quadruped Locomotion