A robot trained in simulation and then deployed for months keeps meeting new physical conditions: a heavier payload, a worn gripper pad, a slicker floor, then back to the original floor. Standard practice handles each shift by collecting fresh real-world data and fine-tuning. That leaves two bad options. The robot can keep every transition it has seen, so storage and training cost grow without bound. Or it can cap its buffer and gradually forget how to behave in conditions it met long ago. It also has to stay safe while it adapts, which is when it knows least about the world. We propose that the robot should not remember what happened. It should remember what the physics was. Each time the world shifts, the robot fits a small number of physical parameters (masses, friction coefficients, actuator gains) to a short window of real data using a differentiable simulator, meaning a simulator whose outputs can be differentiated with respect to its physical constants. It stores only a compact probability distribution over those parameters. When it needs to rehearse old conditions, it regenerates experience by simulating from that stored distribution.
The stored distribution does two further jobs. It is a domain-randomization distribution, the range of simulator settings a policy is trained across, but one that is learned from the robot's own history rather than fixed in advance by an engineer. Its width is also a direct measure of how uncertain the robot is about the current physics, and that width can be turned into a safety penalty. The claim is that one object, a bounded mixture over physical parameters, can serve as the replay memory, the randomization prior, and the uncertainty estimate that keeps exploration safe. Its memory cost grows with the number of distinct physical regimes the robot has met, up to a fixed cap, and not with the number of transitions it has recorded.
Core Claim
For robots whose real dynamics are mostly explained by a parametric simulator, a mixture of Gaussian posteriors over simulator parameters, with a fixed cap on the number of components, can replace the real-transition replay buffer in lifelong safe reinforcement learning. We expect three measurable consequences. Performance on physical regimes the robot revisits should match an agent with an unbounded buffer and clearly beat a fixed-size buffer at equal memory. Constraint violations during each adaptation phase should stay close to what a pessimistic, safety-first zero-shot method achieves. The policy should be less conservative than one trained on a single wide, hand-set randomization range, because the learned mixture concentrates on physics that actually occurs. Where real dynamics fall outside what the simulator can express, the method needs a small residual buffer of unexplained transitions. How large that buffer must be is itself a result: it measures how far the simulator's parametric family is from reality.
Concretely, when a shift is detected, the robot fits parameters $\theta$ (the simulator's physical constants) to a window of real states by minimizing a multi-step prediction loss $L(\theta)$. It then forms a Laplace approximation, which treats the loss surface near its minimum as a Gaussian:
$$\theta^* = \arg\min_\theta L(\theta), \qquad \Sigma = \left(\nabla^2_\theta L(\theta^*) + \epsilon I\right)^{-1}.$$
Here $\Sigma$ is the parameter covariance and $\epsilon$ is a small damping constant that keeps the matrix invertible. The pair $(\theta^*, \Sigma)$ becomes one mixture component. It is merged into an existing component if the KL divergence between them falls below a threshold, and the least-used component is merged into its nearest neighbor once a cap $K$ is reached. With $d$ parameters, memory is $O(K d^2)$, or $O(Kd)$ with a low-rank covariance, and it does not grow with time. Policy training uses the cost of a constrained Markov decision process, a reinforcement learning problem with a separate cost budget alongside the reward. That cost is augmented in the style of SPiDR:
$$\tilde c(s,a) = c(s,a) + \lambda\, \upsilon_{\Sigma}(s,a),$$
where $c$ is the task's safety cost, $\upsilon_\Sigma$ is the variance of simulated next states when parameters are sampled from the component currently believed active, and $\lambda$ is the penalty weight. SPiDR uses a fixed, hand-chosen ensemble. Here the spread comes from how sharply the real data pins down the physics, so it shrinks as the robot identifies a regime and widens right after a shift.
Relation to Prior Work
Several papers motivate the gap. SPiDR turns disagreement across a randomized simulator ensemble into a pessimistic cost bound and reports 100% real-world constraint satisfaction on a Unitree Go1, with standard domain randomization incurring 26.1 times more violations. Its randomization prior is static, and it cannot adapt when the robot's physics drift. We keep its cost-penalty mechanism but derive the spread from identified posteriors that are updated over deployment. Safe Continual Domain Adaptation after Sim2Real Transfer is the closest system-level work. It adapts a randomization-pretrained policy on a KUKA arm with a safety critic and protects old skills with Elastic Weight Consolidation, a penalty on changing weights that mattered for earlier tasks. It raises grasp success from 20% to 60% with zero safety cost during adaptation. It is model-free, it needs a manual trigger to start adapting, and it protects old skills in weight space. We protect them by rehearsing regenerated experience instead, and we detect shifts automatically.
Other work supplies methods. gradSim and SAM-RL show that backpropagating observation error through a differentiable simulator recovers physical parameters. SAM-RL, for example, cuts mass-estimate error from 50% to 20%. Both address a single task and have no safety constraints. We reuse their identification step inside a lifelong loop. Online Structured Laplace Approximations keeps curvature summaries across tasks in network weight space. We apply the same Laplace idea in physical-parameter space, which is far lower-dimensional and physically interpretable.
The continual-learning literature constrains the design. FRCL compresses each task into inducing points, a small set of representative inputs, but its memory still grows by one set per task. We impose a hard cap through merging, and whether merging destroys old regimes is tested directly. Generative pseudo-replay with learned models such as VAEs (Rostami et al., Ye et al.) is our method's direct analogue. The difference is that our generator is a physics engine conditioned on a few numbers, so it cannot itself forget. LEEDS detects task switches from loss spikes in constant memory. The research file notes that such spike heuristics struggle with gradual drift. We detect shifts from simulator prediction residuals instead, and we evaluate gradual drift explicitly.
The safety literature both supplies and constrains. Safe Pontryagin Differentiable Programming keeps every optimization iterate safe, but only under a perfectly known deterministic model. Our setting is the opposite: the model is uncertain by construction. Provably Safe Sim-to-Real Transfer directs real-world data collection to regions where simulator and reality disagree, but it is tabular. Our residual buffer is a continuous, heuristic analogue of its mismatch region, and we make no tabular-style guarantee. Breaking Safety Paradox observes that safer policies generate fewer violations and therefore give the safety critic less to learn from. Regenerated simulated rollouts from stored regimes can supply those counterfactual unsafe transitions without real risk, and we measure whether they do.
Research Program
The system has four parts. First, a differentiable simulator in MJX, the JAX port of MuJoCo in which SPiDR already runs Safety Gym, exposes the physical parameters as differentiable inputs. Second, a shift detector compares one-step and multi-step prediction residuals under the active component against the residuals seen during calibration. When a new window of data is poorly explained by every stored component, it opens a new component. When a stored component explains the data well, it reactivates that one. Third, a constrained learner is run on-policy with simulated data drawn from the stored mixture. We use PPO-Lagrangian with PID-controlled multipliers, following Safety Gymnasium, and FOCOPS (First Order Constrained Optimization in Policy Space) as a second optimizer. Because the replay is simulated, the usual concern that on-policy methods discard their data disappears. Old regimes are rehearsed by sampling components in proportion to recency and usage. Fourth, a bounded residual buffer stores real transitions whose error remains high even under the best-fitting parameters. It feeds a small residual dynamics correction, similar in spirit to SAM-RL's residual network.
The analysis asks three questions. How does forgetting scale with the cap $K$ and with the number of distinct regimes? How much of each regime's real data is explained by parameters and how much must go to the residual buffer? Does Laplace-derived width track the true identification error well enough to justify it as a safety penalty? For the last question we compare the width against held-out prediction error, and we compare violation rates under $\lambda$ values tuned once and then fixed across all regimes.
Baselines
The naive reference point is a policy trained once with wide, fixed domain randomization and never updated. Every other method needs to show it adds something over this point.
The strongest competitor on safety is SPiDR with its original fixed ensemble, since it is the only method in the file with real-robot evidence of full constraint satisfaction under dynamics uncertainty (100% on Go1; standard randomization showed 26.1 times more violations). We run it with the same MJX simulator, the same constrained optimizer, and the same total number of simulator steps, and we give it the same budget for tuning $\lambda$. Our method beats it only if violations stay within a comparable range and return on the current regime is clearly higher, because the learned mixture concentrates on the physics actually encountered.
The strongest competitor on continual adaptation is the SCDA recipe from Safe Continual Domain Adaptation: real-data safe fine-tuning with Elastic Weight Consolidation. We give it our automatic shift detector so it is not handicapped by manual triggering, and we give it the same number of real transitions. The only reported number for this baseline is on the KUKA grasping task (20% zero-shot to 60% adapted success). No reported result found on Lacuna for it on multi-regime sequences in Safety-Gymnasium.
Two replay baselines are matched on memory in bytes. One is a reservoir buffer of real transitions; the other is an unbounded buffer, which serves as an upper reference. A generative-replay baseline trains a VAE-style generator on real transitions at a matched parameter budget.
Two ablations isolate the key ingredients. The first replaces gradient-based identification with CMA-ES, a black-box evolutionary optimizer, using equal simulator compute. Its covariance is estimated from the final population rather than from the Hessian. This tests whether differentiability matters. The second removes merging, so memory is uncapped, and tests whether the cap causes forgetting. Beating a replay baseline means higher mean return on revisited regimes at equal memory, with no more cumulative cost.
Why not just…?
Why not just keep a fixed-size replay buffer of real transitions? This is the most direct competitor, and at small numbers of regimes it may well tie our method. The dilution argument (Mi et al.) predicts that samples per old regime fall toward zero as regimes accumulate. Our sequence-length sweep (4, 16, and 64 regime switches with revisits) is designed to show where the crossover occurs. If a reservoir buffer matches us at 64 switches under equal bytes, the memory claim fails, and we will report it that way.
Why not just randomize widely from the start so nothing needs to be remembered? Wide fixed randomization tends toward conservatism: SCDA's zero-shot grasp policy reached only 20% success before adaptation. Paired with SPiDR-style pessimism it would likely be safe, but the penalty stays large everywhere, including in regimes the robot has already identified precisely. The comparison against fixed-ensemble SPiDR measures exactly this gap in return at matched violations. If the gap is small, the practical argument for adaptation is weak on these tasks.
Why not just identify parameters with a black-box optimizer instead of a differentiable simulator? This alternative is a genuine threat. Contact discontinuities make gradients unreliable (DiSECt needed purpose-built smoothing), and a derivative-free optimizer such as CMA-ES may do well when there are only a handful of parameters. For this reason the core claim is about storing physics parameters as memory, not about gradients. The CMA-ES ablation, together with a sweep over parameter dimension (roughly 4 to 40 parameters, obtained by making friction and mass per-link), tests whether gradients and Hessian-based widths earn their cost. We expect an advantage only at higher dimension and in the smooth-dynamics tasks from the Safe PDP suite, and we will report a null result if that is what we find.
Why not just use Elastic Weight Consolidation as SCDA does? Weight-space penalties compete with plasticity: protecting old weights slows adaptation to new regimes. They also store Fisher information whose number grows with the number of consolidated tasks unless it is merged. The SCDA baseline in our regime sequences directly measures return on revisited regimes and the speed of adaptation to new ones.
Proposed Experiments
The main testbed is Safety-Gymnasium PointGoal2 and CarGoal ported to MJX, following SPiDR's port, and extended with scheduled dynamics sequences. The sequences vary mass, friction, and actuator gain. They include abrupt switches, gradual linear drift, and revisits to earlier regimes, and they include one deliberately out-of-family shift, an unmodeled actuation delay, that the parametric simulator cannot express. The Safe PDP quadrotor and rocket-landing environments add smooth-dynamics cases where gradients are clean. No reported result found on Lacuna for any method on such lifelong regime sequences, so every baseline is rerun under the matched protocol described above.
The metrics are return on each regime at every revisit, which measures forgetting; cumulative cost violations during each adaptation window; the number of real transitions consumed per regime; and memory in bytes over time. We also report the fraction of real data that ends up in the residual buffer. On one task we reproduce the FOCOPS Humanoid-Circle setting as a sanity check on the constrained optimizer. FOCOPS reports a reward of 1106.1 with cost within the limit of 50.
The decisive checks are the following. At 64 switches and equal memory, our method must beat reservoir replay on revisit return. Its violations during adaptation must stay near SPiDR's while its return is higher. Under gradual drift, the residual-based detector must not fragment the mixture into many near-duplicate components. The out-of-family delay shift is expected to flood the residual buffer. If it does not, and the method silently fits wrong parameters and becomes unsafe, that exposes the method's main risk, and we will report it as such.
If the simulation results hold, a stretch test repeats the payload-change sequence on a Go1-class quadruped, carrying and removing a known mass. This is the hardware setting in which SPiDR was validated.
References
- SPiDR: A Simple Approach for Zero-Shot Safety in Sim-to-Real Transfer
- Safe Continual Domain Adaptation after Sim2Real Transfer of Reinforcement Learning Policies in Robotics
- gradSim: Differentiable simulation for system identification and visuomotor control
- SAM-RL: Sensing-Aware Model-Based Reinforcement Learning via Differentiable Physics-Based Simulation and Rendering
- Online Structured Laplace Approximations for Overcoming Catastrophic Forgetting
- Functional Regularisation for Continual Learning with Gaussian Processes
- Generative Continual Concept Learning
- Lifelong Twin Generative Adversarial Networks
- Algorithm Design for Online Meta-Learning with Task Boundary Detection
- Safe Pontryagin Differentiable Programming
- Provably Safe Sim-to-Real Transfer
- Breaking Safety Paradox with Feasible Dual Policy Iteration
- Safety Gymnasium: A Unified Safe Reinforcement Learning Benchmark
- First Order Constrained Optimization in Policy Space
- Continual Learning for Natural Language Generation in Task-oriented Dialog Systems
- DiSECt: a differentiable simulator for parameter inference and control in robotic cutting