Robots trained in simulation increasingly adapt on the fly. A small network looks at the last fraction of a second of joint readings and commands, then guesses the hidden physics, such as how slippery the ground is, how heavy the payload is, or how weak the motors have become. The control policy uses that guess to adjust its behavior. These estimators always look back over a fixed window of history, chosen once before deployment. Theory on learning in changing environments says the right window length depends on how fast the world is changing. A short window is noisy when conditions are steady. A long window lags when conditions shift. Real robots face both regimes at once: ice appears in one step, and motors heat up over an hour. We propose running several estimators that remember over different timescales and combining them online. The weights are set by a signal the robot can actually observe on hardware, namely how well each estimator's physics guess predicts the robot's next measured state.
The opportunity is that a standard tool from online learning fits naturally inside the most successful sim-to-real architecture. That tool is "fixed-share" expert weighting, an algorithm that tracks whichever predictor is currently best and recovers quickly when the best one changes. It gives guarantees without knowing in advance how much the environment will drift. No paper in the research file has connected the two. The practical question is sharp and falsifiable. When the real rate of change differs from what the simulator showed during training, does online timescale selection keep tracking the hidden physics, while a single learned estimator degrades?
Core Claim
Rapid adaptation methods such as RMA: Rapid Motor Adaptation for Legged Robots work in two stages. They first train a policy conditioned on privileged simulator parameters $z_t$. They then train a 1D convolutional module to recover $z_t$ from a fixed-length proprioceptive history. AdaptManip uses a recurrent estimator for the same purpose. The non-stationary learning literature establishes a related fact. Stochastic Multi-Armed-Bandit Problem with Non-stationary Rewards shows that the optimal forgetting rate is a function of the variation budget $V$, the total amount the environment changes over the horizon. Dynamic Regret of Online Markov Decision Processes and Dynamic Regret of Adversarial Linear Mixture MDPs show that an ensemble over exponentially spaced timescales can match an oracle that knows $V$. Taken together, these results imply that any fixed adaptation window is tuned, implicitly, to one variation budget.
Our claim is that sim-to-real adaptation should select its memory timescale online, using a loss that is observable on hardware. Concretely, we train $K$ estimators $\hat z^{(k)}_t$ with history lengths $w_k = w_0 2^k$. We also train, in simulation, a forward model $f(s_t, a_t, z)$ that predicts the next proprioceptive state given the physics latent. At deployment each estimator incurs the loss
$$\ell_{k,t} = \| s_{t+1} - f(s_t, a_t, \hat z^{(k)}_t) \|^2,$$
which needs no privileged labels. A fixed-share master updates weights
$$p_{k,t+1} \propto (1-\alpha)\, p_{k,t}\, e^{-\eta \ell_{k,t}} + \alpha / K,$$
where $\eta$ is a learning rate and $\alpha$ is a small mixing rate that lets weight return to an estimator after a regime change. The policy receives $\bar z_t = \sum_k p_{k,t} \hat z^{(k)}_t$. Standard fixed-share analysis bounds the cumulative prediction loss against the best sequence of timescales in hindsight. That bound holds without knowing $V$, and $\eta$ and $\alpha$ can themselves be set by doubling tricks. We claim this guarantee only for the prediction loss, not for policy return. Turning it into a return guarantee for deep continuous policies is the open problem noted around Natural Policy Gradient for Average Reward Non-Stationary Reinforcement Learning, and we do not attempt it.
Relation to Prior Work
The closest algorithmic relative is BORL-NS-NAC from Natural Policy Gradient for Average Reward Non-Stationary Reinforcement Learning. It treats restart and step-size configurations of a natural actor-critic as bandit arms and selects among them by observed reward, reaching $\tilde O(\Delta_T^{1/6}T^{5/6})$ dynamic regret on synthetic tabular MDPs. We differ in three ways. We select among state estimators rather than among learning algorithms. The selection signal is a dense, per-step physics prediction error rather than sparse, delayed reward. And the policy is frozen, so no learning happens on the robot. The change-detection line, represented by Detection Is All You Need and Efficient Contextual Bandits in Non-stationary Worlds, offers the main alternative mechanism of detecting a shift and then resetting. We include it as a baseline.
On the robotics side, three methods motivate the gap. Sim-to-Real Transfer of Robotic Control with Dynamics Randomization randomizes physics per episode and lets an LSTM adapt implicitly. RMA does explicit latent estimation over a fixed window. Latent Adaptation of Foundation Policies for Sim-to-Real Transfer recalibrates a latent in about six seconds from a batch of target data. All three treat the hidden physics as constant within an episode or calibration batch. None reasons about how fast it drifts.
Two further lines constrain rather than supply the method. SPiDR and Provably Safe Sim-to-Real Transfer handle the reality gap through pessimism and through identifying mismatch regions. Both assume a stationary target world. The second also assumes a discretely separated mismatch, which continuous thermal drift violates. We do not address safety directly, but tracking drift should reduce the time spent acting on stale physics estimates. Contact-Aware Neural Dynamics is a caution: explicit identification of contact dynamics is hard. Our forward model only needs to rank estimators, not to be accurate in absolute terms, which is a weaker requirement we will verify directly.
Research Program
The first component is a drifting-dynamics benchmark. Standard domain randomization draws parameters independently per episode, so it never exercises the failure we care about. We will build simulated environments in Isaac Gym, using an A1-like quadruped and a Franka pushing task, and in the MuJoCo environments of RRLS (code). In these environments, friction, motor strength, payload mass and actuator latency evolve within long episodes under processes with a controlled variation budget $V$. The processes are of three kinds: slow ramps that mimic motor heating, random walks, and piecewise-constant jumps that mimic surface transitions. Sweeping $V$ over two orders of magnitude, and choosing which parameters drift, lets us plot tracking error against $V$. That curve is the object non-stationary theory makes predictions about.
The second component is the estimator pool, the forward model and the master, all trained in simulation. To keep the comparison honest, every baseline estimator is also trained on drifting simulation data with $V$ drawn from a training range. Our method therefore cannot win merely by having seen drift.
The third component is analysis. We will measure whether forward-prediction loss ranks estimators in the same order as true latent error, using rank correlation in simulation, where $z_t$ is known. A low correlation would undermine the whole approach, so this is checked first. We will also check whether master weights shift toward short windows after jumps and toward long windows during calm stretches.
The fourth component is a hardware stage on a quadruped. It uses long continuous runs that heat the motors, combined with transitions between surfaces such as carpet, a lubricated plate and grass, and a payload that is added mid-run.
Baselines
The naive reference point is a domain-randomized policy with no adaptation module, trained with the same PPO budget. Sim-to-Real Transfer of Robotic Control with Dynamics Randomization is the reference for how far randomization alone goes: a randomized feedforward policy reached 0.67 real puck-pushing success, against 0.89 ± 0.06 for the randomized LSTM. Beating this baseline is necessary but not interesting.
The main competitor is RMA with a fixed history window, following RMA. We run it at every window length in our pool. We report both the single window chosen on the training distribution, which is what a practitioner would deploy, and the best window chosen in hindsight separately for each test $V$. The second is an oracle that knows the variation budget, which is exactly the comparator the theory targets. Success means beating the deployed window and coming within a small margin of the per-$V$ oracle across the whole sweep, especially for $V$ outside the training range. RMA reports 100% success on sand, mud and hiking trails, 80% on cement piles and 70% on stairs on a real A1. These are stationary-terrain results on hardware we cannot replicate exactly. On drifting-dynamics benchmarks there is no reported result found on Lacuna for RMA or for any method, so our numbers will be the first.
The strongest learned competitor is a recurrent or long-context attention estimator trained on the same drifting simulation data. It could, in principle, learn to set its own effective timescale. It receives the same data, the same parameter count as our full pool, and the same hyperparameter search budget. We also include a change-detection baseline: the long-window estimator plus a CUSUM-style alarm on forward-prediction error that resets the window to its shortest length. The alarm threshold is tuned on the training range. Finally, we include periodic batch recalibration in the style of Found-adapt, re-run at fixed intervals.
Two ablations isolate the key ingredient. In the first, the master weights estimators by observed reward, as BORL-NS-NAC does, rather than by prediction error. This tests whether a dense, physics-grounded loss matters. In the second, the pool is averaged uniformly with no master. This tests whether online selection matters at all rather than ensembling alone. Metrics throughout are four: cumulative latent estimation error in simulation; the cumulative return gap to a policy given the true $z_t$, which serves as a dynamic regret proxy; falls or failures per hour; and recovery time after a jump.
Why not just…?
Why not just train a recurrent or long-context estimator on drifting simulation data? This is the real threat. If temporally correlated randomization teaches a single network to infer the drift rate from its history, the master adds nothing. Within the training range of $V$ we expect that network to be competitive and possibly better, because it can learn nonlinear forgetting. The decisive experiment is the out-of-range sweep. The network's implicit timescale was fit to the simulated drift distribution, whereas fixed-share selects online with a guarantee that does not depend on $V$. If the learned estimator matches us when tested at $V$ well outside its training range, we will report that correlated randomization is sufficient and the multi-scale machinery is unnecessary.
Why not just tune one window carefully in simulation? The tuned window is optimal only for the simulated variation budget. The support-mismatch problem the file identifies for domain randomization applies to timescales as much as to parameter ranges, and real drift rates are exactly what simulators get wrong. The per-$V$ oracle curve shows directly how much a single window loses as real $V$ moves away from the value it was tuned on.
Why not just detect changes and reset? Detection handles sudden jumps well, and Detection Is All You Need gives optimal prior-free guarantees for piecewise-stationary settings. Slow thermal drift, however, never crosses a threshold cleanly, so detection either fires late or fires falsely and throws away useful history. The ramp-versus-jump split in our benchmark tests this. If detection matches us on both kinds of drift, it is the simpler choice and we will say so.
Why not just fine-tune or re-identify the system on the robot? Online real-world learning is expensive and unsafe. Provably Safe Sim-to-Real Transfer reports 296,910 real samples and 68.5% unsafe iterations for unanchored real-world learning, even in a gridworld. Periodic batch recalibration is included as a baseline to measure what is lost by updating only at intervals.
Decisive Checks
The idea stands or falls on three results. First, in simulation, forward-prediction loss must rank estimators consistently with true latent error. A rank correlation near zero ends the project, because then the master has no usable signal. Second, on the variation-budget sweep, the method's return gap must track the per-$V$ oracle window across the sweep, including out-of-range $V$. Meanwhile the learned recurrent estimator and the deployed fixed window must degrade measurably outside the training range. Third, the uniform-averaging ablation must lose to the full master, which shows that online selection, rather than ensembling, is responsible. Only if these hold do we move to hardware. There we report falls per hour and recovery time across hour-long motor-heating runs with surface and payload transitions. These are compared against RMA-style fixed-window estimation and against the recurrent estimator trained on drifting simulation data.
References
- RMA: Rapid Motor Adaptation for Legged Robots
- AdaptManip: Learning Adaptive Whole-Body Object Lifting and Delivery with Online Recurrent State Estimation
- Stochastic Multi-Armed-Bandit Problem with Non-stationary Rewards
- Dynamic Regret of Online Markov Decision Processes
- Dynamic Regret of Adversarial Linear Mixture MDPs
- Natural Policy Gradient for Average Reward Non-Stationary Reinforcement Learning
- Detection Is All You Need: A Feasible Optimal Prior-Free Black-Box Approach For Piecewise Stationary Bandits
- Efficient Contextual Bandits in Non-stationary Worlds
- Sim-to-Real Transfer of Robotic Control with Dynamics Randomization
- Latent Adaptation of Foundation Policies for Sim-to-Real Transfer
- SPiDR: A Simple Approach for Zero-Shot Safety in Sim-to-Real Transfer
- Provably Safe Sim-to-Real Transfer
- Contact-Aware Neural Dynamics
- Isaac Gym: High Performance GPU Based Physics Simulation For Robot Learning
- Rrls: Robust reinforcement learning suite