Differentiable simulators give a walking controller exact gradients: how much the reward would change if a motor command were nudged. That can make training more than ten times more sample-efficient than trial-and-error reinforcement learning. The catch is contact. Real feet either touch the ground or they don't, and they either stick or slip. Those all-or-nothing events have no useful gradient, so differentiable simulators replace them with smooth approximations. A smoothed normal force pushes up slightly before the foot actually lands. A smoothed friction law lets a planted foot creep under a small tangential force. Our hypothesis is that a policy trained this way learns to use those non-physical forces. When the policy runs on hard contact, in another simulator or on a robot, the borrowed traction disappears and the foot slides.
We propose to make that failure measurable and then remove it. We will treat stance-phase foot sliding, measured under hard contact, as the observable signature of contact exploitation. We will train locomotion policies with a "hard forward, smooth backward" contact model: the simulated physics uses genuine unilateral contact and Coulomb stick-slip friction, while only the gradient is computed from a smooth surrogate. The claim to be tested is concrete. Policies trained this way should keep the sample efficiency of analytically smoothed differentiable training while matching the foot-slip rates and cross-simulator transfer of policies trained with hard contact and domain randomization.
Core Claim
Learning Deployable Locomotion Control via Differentiable Simulation trains an ANYmal quadruped with a sigmoid-smoothed contact model in NVIDIA Warp. It uses Short-Horizon Actor-Critic (SHAC), an algorithm that backpropagates through a few simulation steps and uses a learned value function beyond that horizon. The policy transfers zero-shot to hardware. The same authors, however, name a trade-off: soft contact gives good gradients and transfers poorly, while hard contact does the reverse. They never measure slip, so we cannot tell how close their policy sits to the edge of that trade-off.
Is Linear Feedback on Smoothed Dynamics Sufficient for Stabilizing Contact-Rich Plans? shows the danger in manipulation. Feedback designed on log-barrier-smoothed dynamics "pulls" on objects it is not grasping, which produces 0.30 m terminal error against 0.046 m for open-loop playback. On the surface these two results conflict. We reconcile them with a hypothesis. A four-legged robot is statically stable and forgiving, so it can survive a mild dependence on non-physical forces. A biped balancing on two small contact patches, where ground forces can only push and must stay inside the friction cone, should not. The locomotion analogue of the "pulling" failure is traction the ground cannot actually supply, and foot sliding is where it becomes visible.
The method borrows the straight-through idea from Differentiable Simulation of Hard Contacts with Soft Gradients for Learning and Control (DiffMJX). That work runs realistic hard contact in the forward pass and injects artificial "contacts from distance" (CFD) only in the backward pass, and it uses adaptive time-stepping to suppress gradient oscillations from stiff integration. That work demonstrated the idea on system identification from real cube tosses and on musculoskeletal arm control. It did not test it on legged locomotion or on policy transfer, and it did not apply it to the tangential (friction) direction. Our change is to apply the estimator to both the normal and the friction components at the feet. With $F_{\text{hard}}$ the contact force from a hard-contact solver and $F_{\text{smooth}}$ a smooth surrogate (sigmoid in penetration depth for the normal part, $\mu F_n \tanh(v_t/\epsilon)$ for friction, where $v_t$ is foot tangential velocity and $\epsilon$ a smoothing width), we simulate with
$$\tilde F = F_{\text{hard}} + F_{\text{smooth}} - \mathrm{sg}(F_{\text{smooth}}),$$
where $\mathrm{sg}$ stops gradients. Because $\tilde F$ takes the value $F_{\text{hard}}$ forward and the gradient of $F_{\text{smooth}}$ backward, the policy is never rewarded for forces that only exist in the surrogate.
Feet are spheres or boxes, which sidesteps the >200-vertex mesh scaling problem DiffMJX reports. The repulsive gradient artifacts it reports under complex multi-contact remain a real risk, and we measure them.
Research Program
The first stage is diagnostic and needs no new method. We train the same locomotion task three ways at matched sample budgets: SHAC on sigmoid-smoothed contact reproducing the Deployable Locomotion setup, SHAC on hard contact, and PPO on hard contact with standard dynamics randomization. We then evaluate every policy in hard-contact MuJoCo and at least one further engine, in the spirit of PolySim. For each policy we record two quantities. The exploitation index is the fraction of vertical and tangential impulse, in the training simulator, delivered while the foot either has positive clearance or is moving tangentially below the smoothing width, meaning the force came from the surrogate rather than from contact. Stance slip is the mean tangential foot speed during frames where the hard-contact normal force exceeds a threshold, measured in the evaluation engine. We run this on a quadruped and on a humanoid (Unitree G1 model). The prediction is that exploitation index and slip are low for the quadruped and markedly higher for the humanoid.
The second stage replaces smoothed training with the straight-through contact model above, implemented in MJX with adaptive integration as in DiffMJX, and repeats the evaluation. The third stage adds domain randomization over the smoothing width $\epsilon$ and surrogate stiffness alongside friction, mass and motor gains. The argument is that if any residual exploitation leaks through the backward pass, randomizing the surrogate stops the policy from tuning itself to one particular approximation. This is a cheap, within-engine version of PolySim's multi-engine argument. Randomizing friction coefficients alone cannot fix a structural modeling error such as the wrong friction law, which PolySim's own framing identifies as the limit of parameter randomization. Randomizing the surrogate targets exactly that kind of error.
The foot-sliding literature supplies motivation and a metric, not the method. Kinematic zero-velocity losses such as Combining Recurrent Neural Networks and Adversarial Training for Human Motion Synthesis and Control glue feet in place without any balance or friction-cone reasoning, which Physics Constrained Generative Human Motion Synthesis identifies as a failure. Pipelines such as LAGOON and DynaRetarget remove sliding by forcing motion through a hard-contact simulator using PPO or sampling-based optimization. We adopt their view that hard-contact forward physics is what removes sliding, and ask whether gradients can be kept without giving that up.
Baselines
The naive reference point is SHAC on the sigmoid-smoothed contact of Learning Deployable Locomotion Control via Differentiable Simulation, run unchanged. This is what a practitioner would do today. It defines the sample-efficiency bar, since that paper reports over an order of magnitude better sample efficiency than PPO on its locomotion task. We found no reported absolute return or slip number on Lacuna for it, so we measure its slip ourselves.
The strongest competitor on transfer quality is hard-contact PPO with dynamics randomization and an explicit foot-slip penalty in the reward, as widely used in legged RL and exemplified by the LAGOON pipeline. The slip penalty matters: without it the comparison would be a straw man. For humanoids we add the multi-engine version from PolySim. PolySim reports a 52.8% higher execution success rate on unseen MuJoCo than single-simulator (IsaacSim) training on the 14 ASAP motions. We will rerun it on those motions rather than compare to the reported figure across differing setups.
The strongest competitor on speed is FastSAC/FastTD3 from Learning Sim-to-Real Humanoid Locomotion in 15 Minutes. It trains transferable G1 locomotion in 15 minutes on one RTX 4090, but we found no reported slip or tracking-error number on Lacuna for it.
For fairness, all methods share the robot model, randomization ranges, observation and action spaces, and a reward capped at fewer than ten terms as in the 15-minute recipe. Every method gets the same slip penalty weight grid. Tuning budget is equal numbers of hyperparameter trials, and we report both environment steps and wall-clock time on the same GPU.
Two ablations isolate the key ingredient. The first applies the straight-through estimator to the normal force only, keeping hard friction forward and backward; this tests whether the tangential direction is where exploitation lives. The second removes surrogate randomization.
Beating the baselines means two things at once. Our method must match or undercut hard-contact PPO-with-slip-penalty on evaluation-engine stance slip and cross-engine success. It must also stay within a factor of two of smoothed SHAC's sample count and beat FastSAC on wall-clock time to equal tracking error.
Why not just…?
Why not just add a foot-slip penalty to PPO and skip differentiable simulation? This is the most serious threat. On transfer quality alone it may well tie us. It is our strongest baseline for that reason. The proposal's value then rests on reaching the same slip and transfer with far fewer samples and on explaining why smoothed training fails. If PPO with a slip penalty matches our sample efficiency at equal wall-clock time, the method claim fails and only the diagnostic stands. We will say so.
Why not just shrink or anneal the smoothing, as Parameterized Quasi-Physical Simulators for Dexterous Manipulations Transfer does by softening contact early and hardening it over training? Annealing recovers physical forces only at the end, when the gradients have again become uninformative. The Deployable Locomotion paper frames exactly this smoothness-versus-accuracy trade-off. It is still a plausible competitor, so we run it as an additional baseline with a tuned schedule. If it matches straight-through contact, we adopt the simpler method and report that.
Why not just use fast off-policy RL, since sample efficiency was the point? FastSAC reaches deployable humanoid gaits in 15 minutes and has no smoothing artifacts at all. Its weakness is not demonstrated in the file for locomotion. DiffMJX shows gradient methods winning on high-dimensional muscle control, but not on walking. We therefore treat it as an open empirical question and settle it with the matched wall-clock comparison.
Proposed Experiments
We run velocity-command tracking for the quadruped and a Unitree G1 model, LocoMuJoCo humanoid and quadruped locomotion tasks (LocoMuJoCo), and whole-body tracking of the 14 ASAP motions used by PolySim. Training happens in MJX. Evaluation happens in hard-contact MuJoCo plus one GPU engine not used for training. Metrics are stance slip, exploitation index, velocity and center-of-mass velocity error (LocoMuJoCo's metric that tracks perceived gait quality), cross-engine success rate, environment steps, and wall-clock time. Where a humanoid is available, we measure hardware slip with motion capture or foot-mounted markers on flat ground and on a low-friction surface.
The first decisive check is the diagnostic. Across all trained policies, exploitation index in the training simulator should predict evaluation-engine slip and cross-engine success drop, and the effect should be larger for the humanoid than for the quadruped. If smoothed-SHAC humanoid policies slip no more than hard-contact PPO ones, the hypothesis is wrong and the method is unnecessary.
The second decisive check is the friction-only-hard ablation. If exploitation shrinks only when friction is made straight-through, the tangential direction is identified as the mechanism. That would be the first locomotion-scale evidence for the tangential-contact sensitivity that Exciting Contact Modes in Differentiable Simulations for Robot Learning found only in planar parameter estimation.
The third check is the head-to-head against PPO with a slip penalty and against FastSAC at matched wall-clock time. That comparison decides whether the method is worth using, as opposed to being merely explanatory.
References
- Learning Deployable Locomotion Control via Differentiable Simulation
- Is Linear Feedback on Smoothed Dynamics Sufficient for Stabilizing Contact-Rich Plans?
- Differentiable Simulation of Hard Contacts with Soft Gradients for Learning and Control
- PolySim: Bridging the Sim-to-Real Gap for Humanoid Control via Multi-Simulator Dynamics Randomization
- Combining Recurrent Neural Networks and Adversarial Training for Human Motion Synthesis and Control
- LAGOON: Language-Guided Motion Control
- DynaRetarget: Dynamically-Feasible Retargeting using Sampling-Based Trajectory Optimization
- Learning Sim-to-Real Humanoid Locomotion in 15 Minutes
- Parameterized Quasi-Physical Simulators for Dexterous Manipulations Transfer
- LocoMuJoCo: A Comprehensive Imitation Learning Benchmark for Locomotion
- Exciting Contact Modes in Differentiable Simulations for Robot Learning