Legged robots that walk "blind", using only their own joint sensors and an inertial sensor, now handle mud, stairs and payloads well. They do it by feeding a short history of joint readings into a network that infers what the ground is doing. These systems still run as one policy at one rate, typically tens of hertz. The terrain itself acts on two very different clocks. Friction, slope and payload change over hundreds of milliseconds. A foot punching through soft soil or catching a stair edge is over in a few milliseconds, which falls inside the interval where a 50 Hz policy is simply holding its last command. Splitting control into a slow and a fast loop is the textbook answer. In learning systems it creates two problems. The fast loop is expensive to train with reinforcement learning, and every time the fast loop changes, the slow loop sees a different world.
We propose to solve both problems with one objective borrowed from classical internal model control. The fast loop is given a narrow job: make the robot's measured joint motion match what a nominal forward model predicted the slow policy's command would produce. It is not trained to maximize the locomotion reward. Trained this way, the 500 Hz reflex gets a dense, per-step learning signal over a horizon of only one slow-policy interval, which sidesteps long-horizon credit assignment. The slow policy, in turn, sees dynamics that stay close to nominal regardless of terrain, so the hierarchy is stationary by construction. Our claim is that this split reaches the rough-terrain robustness of the best flat blind policies with less total simulated experience. It also yields a terrain reflex that can be reused under new high-level policies without retraining.
Core Claim
Consider a slow policy running at 50 Hz that issues joint position targets $a$ and receives a slowly updated terrain summary. Consider also a fast reflex running at 500 Hz that adds a bounded torque correction $u_k$ on top of the standard PD torque at each inner step $k$. Let $\hat q_k = f_\theta(s, a, k)$ be the joint positions and velocities that a forward model $f_\theta$ predicts, trained only on nominal terrain (flat, rigid, nominal mass and friction), from the state $s$ at the start of the slow interval. Let $q_k$ be the measured values. The reflex $\rho$ is trained to minimize
$$ J(\rho) = \mathbb{E}\Big[\sum_{k=1}^{K} \lVert q_k - \hat q_k \rVert_W^2 + \lambda \lVert u_k \rVert^2\Big], $$
where $K = 10$ is the number of inner steps per slow step, $W$ weights joints, and $\lambda$ penalizes torque effort. The horizon is only $K$ steps, and the reward arrives at every step, so the reflex's credit-assignment problem is about as easy as reinforcement learning gets.
Three things follow. First, sample efficiency: most terrain randomization can be spent on this cheap, dense problem instead of on the full locomotion reward. Second, stationarity: to the extent the reflex succeeds, the slow policy's transition dynamics are the nominal ones. That is the property Marino et al. could only get by freezing their low-level policies, and that MELA maintained only with smoothing losses. Third, modularity: the reflex's objective never mentions the task, so it should transfer across slow policies.
The reflex has bounded torque authority, so it cannot hide everything. On sand or at a 45 cm drop the robot genuinely has to change its gait. The slow policy therefore also receives a terrain embedding at 10 Hz, computed from the history of residual discrepancies the reflex failed to cancel. What reaches the slow loop is only what the fast loop could not absorb.
Research Program
What the prior work supplies. The proposal builds on three lines of work in different roles.
The contrastive terrain embedding of the Hybrid Internal Model supplies the slow-loop estimator. That work aligns observation histories with successor states and trains without a privileged teacher in 200M frames. We reuse its objective, but feed it reflex residuals rather than raw history, and run it inside a two-rate system; HIM itself is single-rate.
EvoControl motivates the fast loop. It showed that a 500 Hz learned inner controller beneath a 31.25 Hz outer policy beats direct high-rate reinforcement learning. It reported a normalized Ant return of 368 against 100 for the best other baseline, and -25 against -210 on Safety Critical Reacher. EvoControl trains its inner loop on the task reward using evolution strategies with a PD-to-neural annealing schedule, and it was tested only in fixed-physics environments. We replace the task reward with the tracking objective above and test on randomized terrain.
The classical efference-copy work supplies the conceptual mechanism: it compares commanded against actual motion to detect slopes and contacts (Schröder-Schetelig et al., Searock et al.). We turn that detection signal into a learned, closed-loop correction.
What constrains the design. Two results rule out simpler rate splits.
TARC found on a Go1 that longer open-loop holds lost to shorter ones on hardware (TARC-4 beat TARC-10) because model error compounds during the hold. This is exactly the drift our reflex targets, and it argues against simply lowering the policy rate.
Yang et al. needed derivative-free search to train a gait policy over a 500 Hz model predictive controller, and that controller assumes rigid contacts. This argues against a hand-built low-level controller on soft ground.
What gets built. We will build four components in IsaacGym on a Unitree A1/Go1 model.
The first is a forward model $f_\theta$, a small multilayer network. It is trained by regression on nominal-terrain rollouts of a PD-tracking controller over a 20 ms horizon.
The second is the reflex $\rho$, a small network running at 500 Hz. Its inputs are the current discrepancy $q_k - \hat q_k$, the inner-step index, and the slow command. Its output is a clipped torque correction.
The third is a HIM-style contrastive encoder at 10 Hz. It runs over reflex residual histories.
The fourth is a slow PPO policy at 50 Hz.
Training runs in three phases. We first fit $f_\theta$. We then train $\rho$ with short-horizon PPO on the objective $J$ under full terrain and payload randomization, using slow commands sampled from a crude nominal walking policy. Finally, we train the slow policy with the reflex frozen. We will also test a variant that briefly fine-tunes both jointly, to see whether freezing costs anything.
For accounting, we report total simulated seconds of experience across all phases, the forward-model data included, together with wall-clock time on a single RTX 4090. This makes the comparison with HIM's reported budget honest even though the reflex steps ten times faster.
Baselines
The strongest competitor is the Hybrid Internal Model. It is the most sample-efficient blind policy in the file and needs no privileged teacher. It reports convergence in 1 hour on an RTX 4090 with 200M frames, versus 1.28B frames for RMA. On the long-range stair test it reports an average of 176.5 stairs climbed against 75.3 for RMA. We will retrain it from its published description on the same terrain curriculum, domain randomization and reward, with the same GPU and the same hyperparameter-search budget (an equal number of seeds and sweeps). For our IsaacGym rough-terrain survival metric specifically, we found no reported result on Lacuna for HIM, so our own reproduction serves as the reference. Beating it means one of two things. We could match its terrain success and survival using at most half the simulated experience. Or, at equal experience, we could show higher survival on the impact-dominated terrains: step-downs, stair edges and sudden friction drops.
DreamWaQ is the second learned competitor, a variational context estimator trained with adaptive bootstrapping. It reports 95.23% survival against 82.37% for AdaptationNet and 80.92% for EstimatorNet, and we run it under the same protocol. RMA is the fixed two-rate reference: its adaptation module runs at 10 Hz and its policy at 100 Hz, with privileged teacher training. It tests whether any rate split helps or only ours does.
The naive reference is a single 50 Hz PPO policy over proprioceptive history with full domain randomization, holding each PD target for 10 inner steps. The classical reference is a model-based disturbance observer. It uses the same nominal forward model $f_\theta$, but instead of a learned reflex it applies a fixed-gain torque correction proportional to the discrepancy. If this matches our learned reflex, the learning in the fast loop is unnecessary.
Three ablations isolate the key ingredient. In the first, the reflex is trained on the task reward rather than on $J$, keeping the rates and architecture fixed; this is EvoControl's recipe transplanted to terrain. In the second, a same-rate residual policy at 50 Hz is trained on the task reward, in the style of MULE. In the third, our system runs with the reflex outputs zeroed, which isolates the slow policy's own contribution.
Why not just…?
Why not just let HIM's flat policy output PD stiffness and damping as well as targets? A variable-impedance action space gives a single 50 Hz policy some within-hold compliance at almost no cost, and it is the alternative most likely to erase our advantage. It still cannot react to anything within the hold, because the gains are fixed for 20 ms. It also keeps terrain adaptation tangled with the task reward, so it gains neither the dense-signal efficiency nor the stationarity. We will train this variant under the HIM protocol as an extra baseline. If it matches us on both sample budget and impact-terrain survival, the reflex is not earning its complexity and we will report that.
Why not just run a monolithic policy at 500 Hz? EvoControl's results say direct high-rate reinforcement learning handles credit assignment poorly: the best non-EvoControl baseline reached 100 against EvoControl's 368 on Ant. RMA also notes that domain randomization alone pushes policies toward stiff, conservative gaits. We include a 500 Hz PPO run at equal experience only as a sanity check, and we expect it to lose.
Why not just train the fast loop on the task reward, as EvoControl does? This is our central ablation. The tracking objective should win on sample count because its horizon is 10 steps and its reward is dense. It should also win on stationarity, because a task-trained reflex has no reason to keep the slow loop's world nominal. We will measure both, and if the task-reward reflex matches ours, the internal-model objective is not the reason the system works.
Why not just use a classical disturbance observer? Disturbance observers are the classical way to make a plant look nominal, and on rigid ground they may be enough. With fixed gains, however, they tend to overreact on compliant or slipping contacts, where the right correction depends on the contact state. The fixed-gain baseline described above settles this directly. Our hypothesis predicts the gap opens specifically on mattresses, mud-like compliant terrain and low-friction patches.
Proposed Experiments
The main benchmark is the IsaacGym rough-terrain suite used by HIM and DreamWaQ, with 4,096 parallel environments over slopes, stairs, gaps, pits, and deformable or slippery ground, all on the A1/Go1 model. We report survival rate, velocity-tracking error and torque cost against simulated seconds of experience, so every method gets a sample-efficiency curve rather than a single endpoint. We add three stress tests taken from the file: HIM's long-range stair test, a 45 cm step-down (as navigated by PIP-Loco), and payload and dragging-load randomization. Separately, LocoMuJoCo (benchmark, baselines via robfiras/ls-iq) provides friction, inertia and joint-property randomization on a second simulator. That checks that the reflex is not overfitting IsaacGym's contact solver. If time allows, Go1 hardware runs will repeat TARC's push-recovery comparison.
Three checks decide the proposal.
The first is the stationarity check. We measure how far the slow policy's observed transitions drift from those of the nominal model across terrain types, both with and without the reflex. If the reflex does not substantially shrink this gap on rough terrain, the core mechanism fails regardless of the reward numbers.
The second is the transfer check. We freeze the reflex and train a new slow policy for a different command distribution, high-speed tracking up to 2 m/s. We then compare the experience it needs with training a flat HIM policy from scratch for that task. Reuse without retraining is a claim that only a task-agnostic reflex can support.
The third is the efficiency check at matched outcome. This is the simulated experience needed to reach the HIM reproduction's final survival rate, with every training phase counted. We will be satisfied only by a clear reduction that holds across at least five seeds.
References
- Hierarchical RL Using an Ensemble of Proprioceptive Periodic Policies
- Multi-expert learning of adaptive legged locomotion
- Hybrid Internal Model: Learning Agile Legged Locomotion with Simulated Robot Response
- EvoControl: Multi-Frequency Bi-Level Control for High-Frequency Continuous Control
- Using efference copy and a forward internal model for adaptive biped walking
- Learning to Prevent Failure States for a Dynamically Balancing Robot
- TARC: Time-Adaptive Robotic Control
- Fast and Efficient Locomotion via Learned Gait Transitions
- DreamWaQ: Learning Robust Quadrupedal Locomotion With Implicit Terrain Imagination via Deep Reinforcement Learning
- RMA: Rapid Motor Adaptation for Legged Robots
- MULE - Multi-Terrain and Unknown Load Adaptation for Effective Quadrupedal Locomotion
- PIP-Loco: A Proprioceptive Infinite Horizon Planning Framework for Quadrupedal Robot Locomotion
- LocoMuJoCo: A Comprehensive Imitation Learning Benchmark for Locomotion