A Motion Prior as Its Own Safety Net: Safe Exploration for Stylized Legged Characters

Physically simulated characters and legged robots can now learn natural-looking movement without hand-written style rewards. The method that does this, adversarial motion priors (AMP), trains a second network to tell the robot's short motion snippets apart from motion-capture clips and rewards the robot for fooling it. What these systems cannot do is learn a new task in a cluttered environment without crashing hundreds of times along the way. The strongest recent method for crash-free learning, SOOPER, needs a hand-built "safe fallback controller" that can take over whenever the learner gets close to danger. For a high-degree-of-freedom humanoid or quadruped, building that controller by hand is exactly the kind of engineering that AMP was meant to remove.

This note proposes to drop the hand-built controller. We pretrain a command-following AMP skill policy once, in open terrain. We then use it in two roles: as the low level of a two-level controller, and as the fallback, by issuing its own "halt" or "retreat" command. A small learned model predicts what each command will do over the next few hundred milliseconds, and that model decides when the fallback must fire. Because this model works in the low-dimensional space of commands rather than raw joint states, its uncertainty estimates stay tractable. It can also learn how braking distance depends on gait: a cantering quadruped needs more room to stop than a walking one. If this works, a stylized character learns obstacle-course tasks with near-zero training-time collisions. The only hand-specified signals would be a sparse goal reward and a binary collision cost.

Core Claim

The claim is that a style-constrained skill policy supplies a usable safe fallback for free. Safe exploration becomes feasible for high-degree-of-freedom characters when fallback triggering is decided by a learned, uncertainty-aware model of command-level outcomes, rather than by a fixed distance rule or by a Lagrangian penalty that only takes effect late in training.

The mechanism is borrowed from Safe Exploration via Policy Priors. That work uses a probabilistic world model to predict when cumulative cost could exceed the budget and switches to a safe prior at that point. It also trains inside a "Planning MDP", a simulated environment in which imagined rollouts end whenever the fallback would fire, so the policy learns to avoid needing it. SOOPER reports zero training violations on SafetyGym, RWRL and a physical RC car, but its prior is a heuristic or offline controller, and it was evaluated only on low-dimensional systems. Our change is to make the prior a generative motion prior and to move the world model up one level of the hierarchy.

Research Program

The first stage trains a command-conditioned low-level policy with AMP, following AMP: Adversarial Motion Priors for Stylized Physics-Based Character Control. The only task reward is velocity and heading tracking, as in Adversarial Motion Priors Make Good Substitutes for Complex Reward Functions. That paper showed that a velocity-tracking reward plus a style reward yields natural gait transitions and a Cost of Transport (energy used per unit weight per unit distance) of 0.93–1.12, against 1.37–1.65 for hand-designed style terms. The command space includes zero velocity and short backward steps; these become the fallback. The single style-versus-task weight is tuned once here and then frozen. This is the only place any reward balancing happens, which confines the known brittleness of that weight to a task-independent pretraining stage.

The second stage trains a high-level policy that emits a command every $k$ control steps in cluttered scenes. Alongside it we train an ensemble model that maps (proprioceptive summary, local obstacle features, command) to the next macro-state and the collision cost accrued during it. This mirrors SOOPER, but each transition is one command-long macro-step of the frozen worker. Disagreement across the ensemble supplies the epistemic uncertainty.

The fallback fires when either of two things holds. The first is that the pessimistic predicted cost of committing to the proposed command and then halting exceeds the remaining budget. The second is that the model's uncertainty for that command is too high. The high-level policy is optimized in the corresponding Planning MDP.

Three questions are analyzed rather than assumed. First, whether the AMP halt command is actually safe from every gait. We will measure braking distance per gait and velocity, since the fallback guarantee rests on it. Second, whether the model's uncertainty is calibrated on collision events. This matters because Breaking Safety Paradox with Feasible Dual Policy Iteration warns that safe policies starve the safety estimator of violation data. We will log how many near-violation macro-steps the model ever sees. Third, whether freezing the worker removes the hierarchical non-stationarity described in Hierarchical Preference Optimization, and at what cost in achievable behavior.

Baselines

The naive reference point is flat AMP with task reward only and no safety mechanism. It sets the ceiling on unconstrained return and the count of collisions we are trying to remove. It uses the same motion data, simulator, total environment steps, and a tuning budget matched to every other method, measured in hyperparameter trials.

The strongest flat competitor is flat PPO with a Lagrangian cost constraint (PPO-Lag), trained with the AMP style reward. We also run its PID-controlled multiplier variant (CPPO-PID) from SafePO, described in Safety Gymnasium: A Unified Safe Reinforcement Learning Benchmark. That paper reports that PPO-Lag achieves a 98% reduction in safety violations compared to standard PPO, with a 45% drop in total reward, on velocity tasks. These are relative figures on non-stylized agents. We found no reported result on Lacuna for any constrained method on stylized AMP characters, so we measure these baselines ourselves with the same cost budget.

The strongest hierarchical competitor is the same frozen AMP worker with a high-level policy trained by PPO-Lag. This is the ablation that isolates our key ingredient, the learned model-based fallback trigger. A second ablation replaces the learned trigger with a fixed rule: halt when the nearest obstacle is closer than $d$, with $d$ tuned over the same trial budget. This is a hand-designed shield in the spirit of HiSaRL, which guards subgoals with a learned Lyapunov region and reports under 0.3% violations across 10,000 test cases in 2D MuJoCo navigation.

Beating the field means two things together. First, training-time violations must be at least an order of magnitude below the hierarchical PPO-Lag baseline. Second, final goal success must be no lower than the best-tuned fixed-distance shield. We also require that the discriminator's style score and Cost of Transport stay within the range of unconstrained AMP.

Why not just…?

Why not just add a collision penalty to the AMP reward? Quanyi Li et al. show that scalar penalties either accept failures when small or produce stationary "safe but useless" policies when large. Hierarchical Apprenticeship Learning with Application to Quadruped Locomotion shows that balancing coupled locomotion terms by hand is sub-optimal. We include one penalty-swept flat run to confirm that this failure also appears with stylized characters.

Why not just run flat constrained RL (PPO-Lag or FOCOPS) with the style reward? This is a real threat to final performance. It is a weaker threat to our actual claim, because Lagrangian methods satisfy constraints only asymptotically. SOOPER reports that primal-dual and CRPO baselines "frequently violated constraints during early training." If flat CPPO-PID matches our cumulative training violations, the hierarchy and the fallback are unnecessary, and we will report that result.

Why not just use a fixed distance shield that commands "halt"? This is the most dangerous alternative, because the fallback itself is identical to ours. Our argument is that stopping distance depends on gait and speed. A single $d$ must therefore be set for the worst case, which makes the controller over-conservative at a walk and possibly unsafe at a canter. The fixed-$d$ ablation, swept over $d$, directly tests this. If some $d$ matches both our success rate and our violation count, the learned model adds nothing.

Why not just put a control barrier function on the output torques? HiSaRL notes that analytic barriers need closed-form dynamics, which are rarely available for contact-rich legged systems. Clamping torques also ignores the gait cycle. We filter at the command level instead, where the worker's own motion prior keeps corrections natural.

Proposed Experiments

The quadruped testbed retargets the German Shepherd clips from the AMP quadruped paper onto a simulated quadruped, adds a standing reference so that the halt command exists, and places it in procedurally cluttered courses with collision costs. The humanoid testbed trains on locomotion and standing clips from AMASS and uses the same courses. For comparability with the constrained-RL literature, we also run the Safety-Gymnasium navigation and velocity tasks that use the Ant and Doggo agents, with the AMP stage replaced by an unconstrained command policy where no motion data exists.

The metrics are cumulative training-time cost and violations, final goal success, discriminator score, and Cost of Transport. Every result is reported across seeds, with uncertainty.

The decisive checks come in three parts. The first is the violations-versus-success curve against hierarchical PPO-Lag and the swept fixed-$d$ shield; this settles whether the learned trigger matters. The second is a per-gait braking-distance audit; if halting from a canter is not safe, the fallback premise fails, and that failure is informative in its own right. The third is a calibration plot of predicted versus realized macro-step cost on held-out near-collision events, which tests whether the safety-paradox starvation actually occurs. A clear negative on the first check, where the fixed shield matches the learned trigger, would lead us to reframe the contribution as "AMP halt commands are sufficient fallbacks" rather than claim a new trigger.