Many robot controllers run in two speeds. A slow "planner" picks a goal, such as target joint positions, a few dozen times per second. A fast inner loop turns that goal into motor torques hundreds of times per second. Each slow decision is effectively held open-loop until the next one, so the planner's real lookahead is limited by how long that decision stays valid. On hardware, "valid" depends on physics the simulator gets wrong. This proposal makes that dependence measurable and turns it into a control signal.
The standard sim-to-real recipe is domain randomization: train in many simulated worlds whose masses, frictions and delays are randomly perturbed, so the policy cannot overfit one. We propose to also use those randomized worlds as a ruler. Take a state and a slow command, execute it with the fast loop in many randomized worlds, and record how quickly the resulting trajectories spread apart. That spread gives an estimate of how long the command can be held before reality is likely to have drifted away from the plan. We distill this estimate into a small network that runs on the robot and decides when the slow planner must replan. We also claim that a fast inner loop trained under randomization slows this spread, so it directly buys the slow planner a longer trustworthy horizon.
Core Claim
The clearest evidence for the problem comes from TARC: Time-Adaptive Robotic Control. That work trains a policy to output both an action and how long to hold it, with a fixed penalty for each switch. It found that long maximum holds (TARC-10) were best in simulation but failed on a Unitree Go1, while shorter holds (TARC-4) transferred. TARC explicitly leaves open a switch cost that depends on the state or on uncertainty. Systems with fixed rate ratios never face the question at all. EvoControl uses a 31.25 Hz planner over a 500 Hz reflex, and Humanoid-Gym uses a 100 Hz policy over a 1000 Hz PD loop. In both, the ratio is a hyperparameter chosen without reference to how wide the randomization is.
Our claim has two parts. The first is that the hold duration a real robot can tolerate is predictable, before deployment, from how fast randomized simulated rollouts disagree under the actual inner controller. We define the admissible hold for state $s$ and slow command $a$ as
$$\tau^*(s,a) = \max \{\tau : Q_{q}\big(\lVert x^{\xi}_{\tau}(s,a) - \bar{x}_{\tau}(s,a)\rVert\big) \le \epsilon\}.$$
Here $x^{\xi}_{\tau}$ is the robot state after holding $a$ for time $\tau$ in a simulated world with randomized physics parameters $\xi$. The term $\bar{x}_{\tau}$ is the median of that state across worlds. $Q_q$ is the $q$-th quantile over randomization draws, and $\epsilon$ is a tracking tolerance.
The second part is that a learned fast reflex trained under randomization increases $\tau^*$ relative to a fixed PD loop. This gives a concrete reason to learn the inner loop beyond reacting to contact.
Research Program
We build a two-rate controller. The slow policy is trained with PPO and outputs joint-position targets. The inner loop is either a fixed PD controller or an EvoControl-style reflex: it is trained with parameter-space evolution strategies, and its authority is annealed from PD to a neural network. The EvoControl anneal is there to avoid the non-stationarity between levels that the file flags.
Following Sim-to-Real: Learning Agile Locomotion For Quadruped Robots, randomization covers masses, friction, damping and motor strength. It also covers control-loop latency and an actuator model, because that paper found dynamics randomization alone insufficient for dynamic gaits without them.
During training, at sampled states and commands, we roll out $K$ randomized worlds in parallel and compute the spread curve and $\tau^*$. A small "divergence critic" learns to predict $\tau^*$ from proprioception and the command. At deployment, the slow planner holds each command until the critic's predicted $\tau^*$ is reached or the observed tracking error exceeds $\epsilon$, whichever comes first.
The slow policy is trained with this replan rule in the loop. It therefore learns to prefer commands that stay predictable across worlds, not just commands that score well on average. This is where the approach differs from TARC. There, a duration policy trained on average reward can exploit long holds that happen to work across draws in expectation.
The contrast with neighbouring horizon methods is also direct. On Rollouts in Model-Based Reinforcement Learning (Infoprop) stops a learned model's imagined rollouts when accumulated information loss crosses a threshold. We borrow the stop-when-uncertain logic but apply it to physical execution under a real inner controller. Our uncertainty comes from randomized physics, not from a learned model's epistemic error. Optimal Cost Design for Model Predictive Control compensates for a short horizon by reshaping the planner's cost. We instead choose the horizon length itself. OCD-MPC was never tested under wide randomization or on hardware.
The analysis goal is a characterization, not only a method. We want to know how $\tau^*$ scales with randomization width, with latency range, and with the inner controller's gain or learned stiffness. We also want to know whether the sim-predicted $\tau^*$ ranks the hardware survival of fixed holds correctly.
Baselines
The naive reference is a fixed-rate controller in the Humanoid-Gym style. It uses one policy rate over PD, trained with the same randomization, and we sweep the rate ratio so that we report the best fixed ratio and do not pick one arbitrarily. Tuning this ratio by grid search is exactly what practitioners do today. If a well-tuned fixed ratio matches our method, the adaptive part adds nothing beyond tuning.
The strongest existing competitor is TARC, reimplemented on the same randomized simulator with duration caps of 4 and 10 and its fixed switch cost. It has the same PPO budget, the same number of environment steps and the same hyperparameter-search budget as our method. TARC is the only prior method that adapts hold duration and has been transferred zero-shot to a Go1. The file reports that it runs at about 16 Hz during steady walking and rises to about 50 Hz after a push. It also reports that TARC-4 succeeded on hardware where TARC-10 failed. No reported result was found on Lacuna for TARC's numeric tracking error, so we will measure it ourselves. Beating TARC means lower real tracking error or push-recovery failure rate at an equal or lower average control rate.
EvoControl is a third baseline, run both with its original deterministic training and with our randomization added. It reports a normalized return of 368 against 100 for the best non-EvoControl baseline on 500 Hz Ant. We use it to confirm that our reflex training reproduces that advantage before layering the replan rule on top.
Three ablations isolate the key ingredient. The first replaces the quantile spread with mean spread, to test whether conservatism in the tail matters. The second computes $\tau^*$ with latency randomization disabled. The third swaps the learned reflex for PD while keeping the replan rule, which tests the second half of the claim.
Why not just…?
Why not just widen the randomization and let TARC learn shorter holds by itself? This is the most serious threat. If harsher randomization makes long holds unrewarding in simulation, TARC's policy might pick safe durations with no extra machinery. RRLS reports that broad randomization and worst-case training trade nominal performance for robustness, so widening should cost speed and agility everywhere. Our method should shorten holds only where the randomized worlds actually disagree. We test this by sweeping TARC's randomization width and plotting nominal task return against held-out-dynamics failure rate. We then check whether our method lies strictly outside TARC's curve. If it does not, we drop the adaptive rule and report the characterization alone.
Why not just tune the hold length on hardware? Hardware tuning works for one robot and one task. It is costly, though, and it cannot be state-dependent: a push mid-hold needs a short hold, while steady walking tolerates long ones. The fixed-ratio sweep above measures exactly how much a state-dependent rule gains over the best single setting.
Why not just run everything at the fast rate with randomization? EvoControl's 500 Hz results show that monolithic policy-gradient training at high rates fails credit assignment, with a normalized return of 100 against 368 on Ant. We include a monolithic 500 Hz randomized PPO run only as a sanity check.
Proposed Experiments
In simulation we use the 500 Hz MuJoCo tasks from EvoControl, with randomization ranges taken from the RRLS environments (sureli/rrls). The sim-to-sim "reality" follows Humanoid-Gym's validation idea: held-out parameters outside the training ranges, plus a latency the training distribution under-covers. On hardware we reproduce TARC's Go1 protocol, covering straight walking, "run then turn", and lateral pushes delivered mid-hold. The metrics are velocity-tracking error, falls per trial, average control rate and task return. The Franka contact tasks from EvoControl are a secondary platform.
The decisive check is predictive. Before any hardware run, we compute the sim-predicted $\tau^*$ distribution for the TARC-4 and TARC-10 policies. The spread-based ruler must say that holds of 10 exceed $\tau^*$ in the states where hardware failures occur, and that holds of 4 do not. We also require a rank correlation between predicted $\tau^*$ and measured real divergence across held-out sim-to-sim conditions.
A second check tests the claim about the inner loop: $\tau^*$ under the randomized learned reflex should exceed $\tau^*$ under PD by a margin that survives across seeds. If the predictor fails to rank hardware outcomes, the central claim is false regardless of task return. If the reflex does not lengthen $\tau^*$, the second claim is false, and learning the inner loop is justified only by contact reactivity.
References
- TARC: Time-Adaptive Robotic Control
- EvoControl: Multi-Frequency Bi-Level Control for High-Frequency Continuous Control
- Humanoid-Gym: Reinforcement Learning for Humanoid Robot with Zero-Shot Sim2Real Transfer
- Sim-to-Real: Learning Agile Locomotion For Quadruped Robots
- On Rollouts in Model-Based Reinforcement Learning
- Optimal Cost Design for Model Predictive Control
- Rrls: Robust reinforcement learning suite