The Gait Clock as a Baseline: Low-Variance, Safe Learning of When Feet Touch Down

Most legged robots walk on a clock. A contact schedule, meaning a fixed, pre-planned sequence of when each foot is on the ground (stance) and when it is in the air (swing), turns the hard hybrid control problem into smooth optimization that runs in real time. It also makes robots fall when a push knocks them out of phase with that clock. Letting a reinforcement learning (RL) policy decide contact timing fixes the rigidity but breaks two other things. Training becomes noisy, because a single timing choice is followed by foot impacts and occasional falls whose effects swamp the gradient signal. Training also becomes unsafe, because the policy must try bad timings to learn that they are bad. We propose to use the fixed schedule itself as the solution to both problems. At every contact event, a resettable simulator runs two continuations from the same state with the same random disturbances: one where the learned policy chooses the timing, and one where the nominal gait clock does. The difference between their returns, and between their constraint costs, is the advantage signal for the timing decision.

Because the two continuations share the same starting state, terrain, and noise until the moment they diverge, most of the impact noise cancels. The comparison also has a direct safety meaning: the policy is credited only for doing better than a schedule we already trust, and it is penalized whenever it does worse on falls or torque limits. In short, the fixed contact schedule stops being the controller and becomes the reference point that makes learning to deviate from it both stable and safe.

Core Claim

The main source of policy gradient variance for contact-timing decisions is the return noise that comes after the decision: impacts, slips, and rare falls. A counterfactual baseline that replays the nominal schedule from the exact same simulator state, with shared random numbers, removes most of this noise. It works better than a learned critic, which has to fit values across contact discontinuities, and better than a group of the policy's own samples, which carries no safety meaning. The paired advantage for a timing decision $a_k$ made at contact event $k$ in state $s_k$ is

$$\hat A_k = R^{H}_{\pi}(s_k, a_k; \xi) - R^{H}_{0}(s_k; \xi),$$

where $R^{H}_{\pi}$ is the return over the next $H$ contact segments when the policy acts, $R^{H}_{0}$ is the return when the nominal clock acts, and $\xi$ is the shared seed for disturbances and terrain. The same construction with constraint cost $c$ in place of reward gives a paired cost advantage $\hat A^c_k$. The variance of the difference equals $\mathrm{Var}(R_\pi) + \mathrm{Var}(R_0) - 2\,\mathrm{Cov}(R_\pi, R_0)$. The method therefore helps exactly when the two branches are strongly correlated, which is the case for small timing deviations and fails for large ones. That condition can be measured and tested directly.

The new element is not branching, and not the CMDP. It is the choice of the trusted fixed schedule as the paired counterfactual, placed at physical contact events. This yields one quantity that serves three roles: a variance reducer, a feasible fallback, and a cost signal that never runs dry.

Relation to Prior Work

Three lines of work make contact timing adaptive. Learning a Contact-Adaptive Controller for Robust, Efficient Legged Locomotion has a 2.5 Hz value-based agent pick discrete stance/swing primitives over a 500 Hz QP. It reports up to 85% better energy efficiency with about $10^5$ samples, but uses no policy gradient and no cost budget. Non-Gaited Legged Locomotion with Monte-Carlo Tree Search and Supervised Learning searches contact sequences with tree search and a supervised value function, with no safety budget. Dynamic Bipedal MPC with Foot-level Obstacle Avoidance and Adjustable Step Timing shortens stance in 0.1 s steps using an error heuristic and dodges 5 m/s obstacles on a simulated Cassie. These papers motivate the proposal by showing that timing freedom pays off. None of them learns timing through policy gradients under a cost constraint, which is the setting we target.

Two papers constrain the design. Evaluation of Constrained Reinforcement Learning Algorithms for Legged Locomotion shows that constrained RL on ANYmal blows up unless cost advantages are normalized; their N-P3O fixes this with global normalization inside a fixed gait. Breaking Safety Paradox with Feasible Dual Policy Iteration shows that as policies get safer, cost samples vanish and the cost critic degrades. Our paired estimator addresses both. It needs no cost critic. And by adding extra "probe" branches that try perturbed timings, it keeps sampling near-violations without the main rollout ever executing them. This idea borrows the logic of OFFER: Off-Environment Reinforcement Learning, which over-samples rare catastrophic events and corrects the bias by importance weighting. We apply that logic to action deviations at contact events rather than to environment variables.

Two papers supply the estimator. Segment Policy Optimization assigns credit to segments using Monte Carlo branches cut at low-probability tokens, but only for language models. Self-critical training (Rennie et al.) uses a greedy rollout as a critic-free baseline. We combine the two: cutpoints come from physics (touchdown and liftoff), and the baseline branch is a known-safe controller rather than the policy's own greedy mode.

On safety, Safe Reinforcement Learning for Legged Locomotion switches to a recovery policy using a centroidal-model reachability check. It reports 48.6% fewer simulated falls than Recovery RL and PCPO, but leaves timing periodic and does not analyze the off-policy gradient distortion that switching introduces. Constrained Markov Decision Processes via Backward Value Functions keeps training feasible through conservative state-wise bounds learned by TD learning, which the mode switches of variable contact make hard to fit. Our version of state-wise safety is relative and estimated empirically: deviate only where the paired cost advantage is confidently non-positive.

Research Program

The system has three layers. The bottom layer is a convex MPC plus QP whole-body controller that tracks a contact schedule; it is the same kind of stack used by the contact-adaptive and safe-legged-RL papers. On top of it sits a nominal trot clock with a fixed period. The learned policy fires at each touchdown or liftoff event. It outputs a bounded timing deviation for the next stance or swing of each leg and a foothold offset, with zero deviation meaning "follow the clock." At each decision event during training, the simulator state is snapshotted. We roll out the policy branch and the nominal branch for $H$ contact segments under a shared seed, and in a fraction of events we also roll out $M$ probe branches with perturbed timings.

The policy update is a FOCOPS-style step (First Order Constrained Optimization in Policy Space). It solves for a reweighted target policy using reward and cost advantages, then projects back to the parametric policy. Here the advantages are the paired $\hat A_k$ and $\hat A^c_k$ instead of critic-based GAE estimates. Probe branches feed the cost estimates through importance weights. The safety gate at execution time is simple: if a running upper confidence bound on $\hat A^c$ for the proposed deviation in similar states is positive, the robot executes the nominal timing instead. Because the nominal branch is computed anyway, this gate introduces no off-policy mismatch.

Analysis runs alongside the method. We will measure gradient variance per contact event and the empirical correlation between paired branches. We will also test the predicted breakdown point: variance reduction should shrink as deviations grow and branches decorrelate. The work will report this frontier honestly rather than only the regime where the method wins.

The method assumes a resettable simulator. Hardware is used only for zero-shot deployment tests, not for on-robot training, and training-time safety claims are about simulated training rollouts.

Baselines

The naive reference is the fixed-clock controller itself, run with the same MPC and QP stack. It anchors every metric; a method that cannot beat it on recovery and dodge success has learned nothing useful.

The strongest learned competitor is N-P3O from Evaluation of Constrained Reinforcement Learning Algorithms for Legged Locomotion. It is the method shown to stabilize constrained RL on a real quadruped, reporting 0.49 violations per episode versus over 533 for unconstrained PPO on ANYmal. We give it the same timing-plus-foothold action space, the same cost functions and budgets, and the same total simulator steps. That last point matters: our branches consume steps, so N-P3O receives correspondingly more on-policy rollouts. Both methods get the same hyperparameter search budget. Beating it means lower violations during training and at convergence together with equal or higher task success at matched simulator steps. No reported result was found on Lacuna for N-P3O with a learned contact-timing action space, so we produce that number ourselves.

The second competitor is the recovery-switch scheme of Safe Reinforcement Learning for Legged Locomotion. It runs on the same robot model, with its reachability trigger and a pre-trained recovery policy, and its learner is allowed variable timing. Beating it means fewer training falls without its model-based trigger, especially under foot slip, where its centroidal model is known to be weak.

The non-learning reference is the adjustable-step-timing heuristic of Dynamic Bipedal MPC with Foot-level Obstacle Avoidance and Adjustable Step Timing, reimplemented on our stack for the obstacle task. On their Cassie setup it dodged obstacles at 5 m/s.

Three ablations isolate the key ingredient. The first keeps branching but uses the mean of $K$ policy samples as the baseline, in the style of GRPO. This tests whether a group baseline does as well as the nominal-clock baseline. The second keeps the nominal baseline but drops shared random numbers, which tests whether variance reduction comes from the pairing. The third removes probe branches, which tests whether they prevent the safety-paradox regression.

Why not just…?

Why not just condition the critic on contact phase and normalize advantages per phase? This is the cheapest fix to the contact-discontinuity problem, and it is a real threat. N-P3O's global normalization already helps a great deal, and per-phase statistics might capture most of the remaining gain at zero simulator overhead. We include it as a baseline: N-P3O with a phase-conditioned critic and per-phase normalization. If it matches paired rollouts on gradient variance and violations at matched simulator steps, the branching machinery is not justified, and we would report that. Our expectation that it falls short rests on the safety paradox documented in Breaking Safety Paradox with Feasible Dual Policy Iteration. Phase conditioning does nothing to stop cost samples from disappearing as the policy improves. The probe ablation should reveal whether that difference matters in practice.

Why not just shape the reward with fall and torque penalties and run PPO? The ANYmal evaluation shows unconstrained PPO incurring over 533 violations per episode and accelerating into a 20 cm drop, while large penalties drive policies toward standing still (Li et al.). We include PPO with shaped penalties, swept over three penalty scales, to show this tradeoff on our timing tasks.

Why not just use a group of the policy's own samples as the baseline, as GRPO and SPO do? This would reduce variance without needing a nominal controller. It is precisely the first ablation. Our prediction is that it reduces variance less, because the policy's own samples decorrelate faster than a small deviation compared against the clock. It also offers no safe fallback for the gate. If it matches on both variance and violations, the specific role of the fixed schedule is not supported.

Why not just plan timing online with MIQP or tree search and skip learning? The cited planners pay with low replanning rates (MIQP at 50 Hz, tree search at 12.5 Hz) and coarse timing grids, and they enforce no cost budget. The push-recovery task under impulses applied mid-swing is designed to probe exactly this latency gap. The adjustable-timing heuristic is run alongside so the comparison is direct.

Proposed Experiments

All training uses the Unitree A1 model from LocoMuJoCo (codebase at loco-mujoco). The constraint costs are falls, torque and joint limits, and foot–obstacle contact, each with a fixed budget. Three tasks span the reasons timing matters. The first is push recovery from lateral impulses at random gait phases. The second is a split-belt treadmill with paused or reversed belts, following the contact-adaptive paper. The third is approaching obstacles at increasing speed, following the Cassie setup.

Metrics are the success rate on each task, cumulative constraint violations over training as well as at convergence, cost of transport, and, as the mechanistic metric, the empirical variance of the policy gradient at each contact event alongside the correlation between paired branches. Every curve is plotted against total simulator steps, with branches included.

Three outcomes decide the proposal. The first is variance: paired advantages should cut gradient variance substantially relative to both N-P3O and the policy-sample-group ablation. The reduction should track the measured branch correlation and shrink at large timing deviations. If variance does not fall, or falls but does not follow correlation, the mechanism is wrong. The second is the safety paradox: late in training, the no-probe ablation should show cost-estimate collapse and violation spikes that the full method avoids. The third is compute-matched utility: at equal simulator steps, the full method should reach higher push-recovery and dodge success than phase-normalized N-P3O while incurring fewer cumulative training violations. If phase-normalized N-P3O matches it, we conclude the cheaper fix suffices.

A final zero-shot transfer of the converged policy to a physical A1 under lateral pushes checks that the learned timing deviations survive outside simulation. It makes no claim about safety during on-robot training.