Legged robots that learn to walk with reinforcement learning tend to settle on one safe gait, usually a trot, and keep using it even when the terrain calls for something else, such as a jump over a gap or a high-stepping climb. Maximum entropy reinforcement learning is the standard remedy for this kind of premature commitment. It rewards the learner for keeping its choices random where randomness costs little. In a robot, however, randomness at the wrong place is dangerous. Noise in joint commands disrupts foot-contact timing that has to be accurate to the millisecond. We propose that, in a two-speed control hierarchy, the randomness should live almost entirely in a slow, temporally committed choice of gait mode and be squeezed out of the fast joint-level controller. The modes themselves should be discovered rather than designed by hand.
Concretely, we would build one fast low-level policy that takes a discrete mode index plus a few continuous style parameters as input. While it trains, a stand-in high level picks modes uniformly at random and holds each choice for a minimum duration, which is the most random high-level behaviour possible. Distinct gaits are encouraged by two terms: a reward for modes that can be told apart from their foot-contact patterns, and an energy-efficiency penalty. We then freeze this policy and train a slow high-level controller with entropy-regularized RL over modes. The test is whether this yields specialist-level terrain performance without specialists designed by hand, and whether the asymmetric entropy allocation is what makes it work.
Core Claim
The claim is specific and falsifiable, and has three parts. First, a single mode-conditioned controller trained this way will produce several qualitatively distinct gaits, measured by clusters of contact patterns. A flat energy-regularized policy produces only the speed-indexed gaits it already finds. Second, on procedural stairs, stepped blocks and bumpy terrain, the resulting hierarchy will match MTAC's hand-partitioned experts without being told which terrains need which skills. Third, moving entropy from the high level into the low level, as a symmetric dual-level soft hierarchy does, will degrade success in a predictable way: low-level temperature should hurt monotonically, while high-level entropy should help on terrains where the best gait only pays off late.
Prior Work and Where This Sits
Three papers motivate the gap. MTAC trains three PPO experts in three separate hand-built worlds (bumpy, stair, step) and adds a gating network over them. It shows that a single generalist PPO policy hits a "specialization ceiling", but its skill set is fixed by the designer's choice of terrains. Adaptive Energy Regularization shows that walk, trot and fly-trot emerge from a single flat policy when it is penalized for cost of transport, which is energy used per unit distance. These gaits are indexed by speed, not by terrain, and the policy has no mechanism for choosing a non-default gait for an obstacle. Discovery of skill switching criteria learns five gaits with SAC (Soft Actor-Critic, the standard maximum entropy actor-critic). Each gait gets its own foot-contact reward, and switching comes from thresholds tuned offline by an evolution strategy. That paper also reports that naive blending of gait distributions produces "mushy" unstable motion and that high-speed transitions are brittle.
Two papers supply the methods and also constrain the design. SHIRO puts SAC at both levels of a goal-conditioned hierarchy and learns both temperatures. Its own analysis requires the worker policy to change slowly, and it reports that a high worker temperature makes the manager's environment too noisy to learn from. We take that warning as a design rule: we train the low level first and freeze it, and we keep its temperature near zero. SOAC shows that soft options collapse to one option, or switch at every step, unless two terms are added. One is a mutual-information term that keeps options distinct; the other penalizes erratic switching. We reuse both ideas, in the forms of the contact-pattern discriminator and the minimum commitment duration, but ground them in gait physics instead of abstract MuJoCo tasks. Soft Q-learning supplies the core argument that entropy keeps multiple high-reward modes alive. Its Ant-maze result was obtained on a flat controller without contact realism.
HRL-CPG is the nearest structural precedent. It places SAC at the high level and a deterministic oscillator underneath, but its authors state that it cannot switch between distinct gaits. Our low level is a learned policy with discrete mode slots precisely so that switching is possible. Latent-interface hierarchies such as LeVERB, Gaitor and Next Steps learn continuous gait or action spaces from demonstrations. They show that a richer interface than velocity commands helps, but they cannot discover gaits absent from the data. Their continuous interfaces are also the kind of unimodal bottleneck we ablate against.
Research Program
The low-level controller runs at 50 Hz in IsaacGym on a Unitree-class quadruped. It observes proprioception, a local height map, a commanded planar velocity, a mode index $k \in \{1,\dots,K\}$ with $K=8$ (deliberately more slots than we expect to need), and two continuous style scalars. During this first stage the high level is replaced by a sampler. It draws $k$ uniformly, holds it for at least 0.5 s, and draws style and velocity at random. The low-level reward is
$$r^{l} = r_{\text{track}} - \lambda\, \mathrm{CoT} + \beta \log q_\phi(k \mid c_{t-T:t}),$$
where $r_{\text{track}}$ is velocity tracking, $\mathrm{CoT}$ is cost of transport as in the energy-regularization work, $\lambda$ and $\beta$ are weights, and $q_\phi$ is a small classifier that predicts the mode $k$ from the last $T$ steps of binary foot-contact signals $c$. The last term pays the controller for modes that look different in their footfall patterns, which is where gaits actually differ. Making modes distinguishable at the contact level, rather than in joint space, is what keeps them from becoming trivially different noise.
The low-level actor uses a small fixed target entropy, and we report its actual action entropy. Terrain during this stage is a randomized mix of all geometries, so no terrain is ever assigned to a mode. Mode slots that never become distinguishable are allowed to merge, and we report how many survive.
In the second stage we freeze the low level. A high-level controller runs at 5 Hz with discrete-action SAC over $k$ and Gaussian outputs for the style scalars. It observes the height map, proprioception and the task command, and it may change mode only at commitment boundaries. The structural claim is that all exploration over how to move happens here, in choices that last half a second, where a bad choice costs a stumble rather than jitter in every step.
We would also build a "deceptive curriculum" variant of the terrain. Low stairs are traversable by trotting and reward it early; tall stairs and wide gaps require a mode that pays off only later. This variant directly tests the premature-commitment hypothesis.
Baselines
The naive reference point is a flat PPO policy trained on the same mixed terrains with domain randomization, the same compute and the same tuning budget. This reproduces MTAC's "Generalized PPO". On MTAC's terrains that baseline is reported at 30% low-speed and 13% high-speed (1.75 m/s) stair completion, 41% on stepped terrain and 13% on high-speed bumpy terrain (MTAC). Our reimplementation should land near those figures before we trust any comparison.
The strongest existing method on this question is MTAC itself. It has hand-partitioned experts and a PPO gate, and is reported at 100% low-speed stairs, 82% high-speed stairs, 83% stepped and 34% high-speed bumpy terrain. We would reimplement its terrains and its three-expert pipeline in the same simulator. Its experts would receive the same total environment steps as our two stages combined, and every method would get the same hyperparameter-search budget. Beating MTAC means matching it within the seed-level confidence interval on stairs and stepped terrain without terrain-specific training worlds, and exceeding it on high-speed bumpy terrain and the deceptive curriculum, where a fixed expert set has no matching specialist.
The flat energy-regularized policy of Adaptive Energy Regularization is the strongest "no hierarchy" competitor, because it already produces multiple gaits. We found no reported result on Lacuna for it on stairs, steps or bumpy terrain, so we would run it ourselves with matched budget.
SHIRO-style symmetric soft hierarchy, with both levels SAC, both temperatures learned and joint training, is the competitor for the entropy-allocation claim. SHIRO reports reaching goals two to four times faster than HIRO on AntMaze-type tasks, but we found no reported result on Lacuna for it on contact-rich terrain locomotion.
Four ablations isolate the key ingredients:
- the full system with the low-level target entropy raised to SHIRO-like levels;
- the full system with a PPO high level and a small fixed entropy bonus instead of soft RL;
- removal of the contact-discriminator term;
- replacement of the discrete mode slot with a single Gaussian latent (a LeVERB-like continuous interface), which tests whether averaging between modes hurts on terrains with two viable but incompatible gaits.
Why not just…?
Why not just use MTAC with its PPO gate, since a categorical gate is already multimodal? This is the most serious alternative. MTAC's gate does choose among modes without averaging them, and its 82% high-speed stair result shows that gating works when the experts are right. Our argument is not that the gate collapses. It is that the experts must be designed per terrain, which fails when terrain needs a skill nobody anticipated. The deceptive curriculum and held-out mixed-terrain courses settle this. If MTAC matches us there, the honest conclusion is that hand partitioning is sufficient and our contribution reduces to automation. We would report it that way.
Why not just train a flat policy with an energy penalty, since gaits emerge on their own? Gaits do emerge that way (Adaptive Energy Regularization), but they are indexed by speed. Nothing in a flat policy lets it deliberately pick a pronk at trotting speed for a gap. MTAC's generalist ceiling (13% on high-speed stairs) is evidence against flat policies, though not against energy-regularized ones specifically. Our comparison on matched terrains is the direct test. If the flat energy-regularized policy comes within a few points on high-speed stairs and stepped terrain, the hierarchy is not justified.
Why not just run SHIRO, which already learns both temperatures? SHIRO's own analysis and its observation about noisy workers suggest that symmetric entropy is fragile. Locomotion is less forgiving than AntMaze, as the brittle high-speed transitions reported in Discovery of skill switching criteria show. Still, learned temperatures might find a low worker entropy by themselves. The low-level temperature sweep and the SHIRO baseline will show whether explicit allocation and staged training buy anything beyond automatic tuning.
Why not just use PPO's ordinary entropy bonus at the high level? For a discrete mode choice this may well be enough, and the PPO-high-level ablation tests exactly that. If it matches soft RL, we would state that the gain comes from the uniform, committed mode sampling used during low-level training together with the contact discriminator, not from soft RL at the top.
Proposed Experiments
All experiments run in IsaacGym on MTAC-style procedural stairs, stepped blocks and bumpy height fields at low speed and at 1.75 m/s. To these we add the deceptive curriculum and randomized low-friction patches, which act as a simulated stand-in for the deceptive terrain studied by AMCO. We use five seeds per method. The multi-task tooling in MTBench serves as scaffolding.
The metrics are:
- success or completion rate;
- falls per kilometre;
- cost of transport;
- number of distinct gaits, counted by clustering contact patterns and reporting how many surviving mode slots map to separable clusters;
- switch frequency;
- base angular-velocity energy as a proxy for jitter.
Three checks are decisive. The first is gait discovery. Does the stage-one controller yield at least one non-trot mode, such as a bound or pronk, that the high level selects on gaps and tall stairs and never on flat ground, where the flat energy-regularized policy produces none? The second is the MTAC comparison. Do we match 82% on high-speed stairs and 83% on stepped terrain without terrain-specific training, and beat 34% on high-speed bumpy terrain? The third is the allocation curve. Raising low-level target entropy should reduce success and increase jitter monotonically, while removing high-level entropy should hurt specifically on the deceptive curriculum and not on uniform terrain.
If the first check fails, the discovery mechanism is wrong. If the second fails while the first passes, the discovered modes are not useful. If the third check shows no asymmetry, the central entropy-allocation claim is refuted. A short sim-to-real transfer of the frozen hierarchy to a quadruped is a stretch goal that comes only after these three checks pass.
References
- MTAC: Hierarchical Reinforcement Learning-based Multi-gait Terrain-adaptive Quadruped Controller
- Adaptive Energy Regularization for Autonomous Gait Transition and Energy-Efficient Quadruped Locomotion
- Discovery of skill-switching criteria for learning agile quadruped locomotion
- SHIRO: Soft Hierarchical Reinforcement Learning
- SOAC: The Soft Option Actor-Critic Architecture
- Reinforcement Learning with Deep Energy-Based Policies
- Hierarchical reinforcement learning with central pattern generator for enabling a quadruped robot simulator to walk on a variety of terrains
- LeVERB: Humanoid Whole-Body Control with Latent Vision-Language Instruction
- Gaitor: Learning a Unified Representation Across Gaits for Real-World Quadruped Locomotion
- Next Steps: Learning a Disentangled Gait Representation for Versatile Quadruped Locomotion
- AMCO: Adaptive Multimodal Coupling of Vision and Proprioception for Quadruped Robot Navigation in Outdoor Environments