Keeping the Still-True Past: Transition-Level Replay Validity for Reset-Free RL Under Drift

A robot that learns without human resets has to live with its own history. It practices a task, then uses a second learned behavior to put things back so it can practice again, and it trains both behaviors from a replay buffer, a memory of past transitions reused many times for efficiency. Now let the world change slowly: a door hinge stiffens, table friction drops, a motor weakens. An episodic agent can shrug this off, because a human reset and fresh data eventually dilute the stale memories. A reset-free agent cannot. Its buffer keeps teaching it how the world used to behave. When the reset behavior learns from outdated physics, it stops returning the robot to useful states, and the agent strands itself. That strands the whole learning loop, and the only way out is the manual intervention the method was built to remove.

We propose to score every stored transition by whether it is still true, meaning consistent with the current dynamics, and to weight replay accordingly. We do not throw away data by age, and we do not flag a single global "the environment changed" event. The signal comes from a small ensemble of one-step dynamics models fitted to a recent window. It uses no reward, so it works in sparse-reward manipulation where reward-based change detectors see nothing. It also uses the structure of reset-free learning: the reset behavior repeatedly drives the robot back through the same states, so the recent window naturally re-covers the regions where old and new data overlap. The bet is simple. Most drift is local. Friction changes contact transitions, not free-space arm motion. Keeping the unaffected past at full weight should preserve rare successes that a recency-only buffer discards, and removing contradictory targets should keep the value estimates from drifting upward along an uninterrupted trajectory.

Where this sits

Reset-free methods pair a forward task policy with a backward reset policy. EARL formalizes the setting and benchmark. MEDAL trains the reset policy to match demonstration state distributions. RISC switches controllers using a learned success critic and bootstraps through switches instead of treating them as terminations. All three assume stationary dynamics, and that assumption is the one we remove. MoReFree and LiSP add learned models to the loop. LiSP tests shifting goals, not shifting physics. MoReFree's reported failure on Sawyer Door comes from contact dynamics its world model could not predict. That failure constrains our design: we use the dynamics model only as a local validity judge over transitions, never as a generator of imagined data.

The closest work on non-stationarity is DEER. DEER shows that standard prioritized replay confuses learning progress with environment change. Its fix is to detect regime changes with a Jensen–Shannon test on reward windows and then prioritize pre- and post-shift data. DEER motivates our proposal, and we differ from it in three ways. It needs episodic resets. Its detector is reward-based and so is blind under sparse reward. Its decision is global over time, whereas ours is made per transition across state space. Fast and slow successor features with synaptic consolidation constrains us from the other side. Under gradual mass drift, it finds that plasticity-injecting weight resets collapse, and that the real bottleneck is keeping still-valid competence. We accept that diagnosis but act on the data rather than the weights. That leaves us free to use Adam, which their consolidation scheme cannot. STAR triggers adaptation from accumulated prediction surprise in low-dimensional games. We borrow the surprise principle and apply it to continuous control, with per-transition granularity.

The value-stability side of the problem supplies motivation and a measurement lens rather than the method. Tsitsiklis's analysis (Analysis of TD with function approximation) and the emphatic-TD study (Should all TD learning use emphasis?) show that bootstrapped targets combined with nonlinear approximators can diverge. EARL's practice of cutting the bootstrap chain every 1,000 steps shows that continuing reset-free learning already needs ad hoc protection. Stale transitions after a drift add another source of mutually inconsistent targets on top of this. We therefore treat value blow-up as a primary outcome, not a footnote.

Core Claim

In reset-free off-policy learning under dynamics drift, weighting replay by per-transition consistency with current dynamics beats both uniform replay and recency-based forgetting when the drift is spatially localized. It should roughly tie with recency-based forgetting when the drift is global. The gain should show up in three places: fewer manual interventions, faster recovery of task success after a shift, and bounded value estimates.

Research Program

The base learner is RISC, unmodified apart from its replay sampling. Alongside it we maintain an ensemble of $E$ small one-step dynamics networks, refit continually on the most recent $W$ transitions. For a stored transition $i$ with prediction error $e_i$, we take the ensemble-mean squared error of $s'$ given $(s,a)$ and compare it to the ensemble's error $\bar e_{\text{recent}}$ on held-out recent transitions whose states lie near $s$. The validity weight is

$$w_i = \min\!\left(1,\ \exp\!\left(-\frac{e_i - \bar e_{\text{recent}}}{\tau\,\sigma_{\text{recent}}}\right)\right),$$

where $\sigma_{\text{recent}}$ is the spread of recent errors and $\tau$ is a temperature. A transition that the current-physics model explains as well as fresh data keeps weight one. A transition it cannot explain fades.

Two caveats need handling. First, the ensemble cannot judge regions the recent window has not visited, and there disagreement among the ensemble members is high. In those regions we freeze the weight at its last value rather than guess. The forward–backward cycle revisits the start region often, so this freezing should rarely last long in the states that matter for resets. Second, weights are recomputed lazily on a rotating slice of the buffer each step, so the added cost is a few forward passes per update. Replay samples are drawn in proportion to $w_i$, and the same weights multiply the critic loss so that invalid targets do not enter bootstrapping. Nothing is deleted, which means a transition can regain weight if the dynamics drift back.

The analysis has three parts. We compare the time a drift occurs with the time the weights respond. We look at which transitions get down-weighted, expecting contact phases to be affected and free motion not. We also track critic magnitude along the lifetime.

Baselines

The naive reference is RISC with its standard uniform replay run straight through the drift. This tells us how badly stationary reset-free learning actually fails. It is essential, because if RISC barely degrades, the problem is not real.

The strongest simple competitor is RISC with recency-biased replay: a FIFO buffer whose capacity is swept over a grid, plus an exponential age-decay variant. It is the most likely to match us, because under global drift, discarding old data is close to the right answer. It will receive the same hyperparameter budget as our method: an equal number of tuning runs over its capacity or decay rate as we spend on $\tau$ and $W$.

The strongest published method is DEER, ported onto the RISC learner with its Jensen–Shannon reward detector and discrepancy-based prioritization. It gets the same replay ratio, network sizes, and tuning budget. The file reports DEER's gain over prior replay methods as an average of 11.54%, rising to 22.53% under extreme shifts (DEER), but only in episodic MuJoCo with resets. No reported result was found on Lacuna for any replay method in reset-free learning under dynamics drift. For RISC on stationary EARL, the file reports only that it "matched or exceeded the best-known results on 3 out of 4 EARL tasks," with no per-task percentages. MEDAL's 40% on Sawyer-Peg is the only specific stationary success figure. Our own stationary RISC runs will therefore serve as the pre-drift reference.

We also include an oracle that knows the exact drift times and flushes the buffer at each one. This bounds what global regime-switching could achieve.

Beating these baselines means lower cumulative manual interventions and higher post-drift success than tuned recency replay under localized drift, at matched compute, with non-overlapping confidence intervals over at least ten seeds. Under global drift we predict a tie, and a tie would not count against the method.

Two ablations isolate the key ingredient. The first uses the same dynamics detector but applies its output globally, down-weighting all pre-change data once a change is flagged. This separates per-transition validity from better change detection. The second swaps our detector for DEER's reward detector while keeping everything else fixed, which isolates the value of a reward-free signal.

Why not just…?

Why not just use a small FIFO buffer or recency weighting? This is the real threat, and we say so. When the whole dynamics shift at once, forgetting by age is close to optimal, and our method should add little. The argument against it is specific to reset-free learning under local drift. Successes on hard tasks like Sawyer-Peg are rare, and EARL reports that autonomous methods other than demonstration-guided ones make little progress there, so a short buffer that ages out those successes throws away the scarcest signal. The localized-versus-global drift comparison settles this directly. If recency matches us under localized drift too, the per-transition idea is wrong and we will report it that way.

Why not just run DEER inside a reset-free agent? DEER's detector watches reward windows. In sparse-reward manipulation, and during the reset phases where no task reward is collected at all, it has nothing to watch. Its prioritization also treats time as the unit, so contact and free-space data are down-weighted together. The detector-swap and global-weighting ablations test each of these limitations separately.

Why not just inject plasticity with periodic network resets? The successor-feature study (Fast and slow successor features) reports that TD3 with plasticity resets collapsed on Humanoid under severe mass drift. In a reset-free agent the damage is worse, because wiping the reset policy's competence can leave the robot unable to recover at all. We include shrink-and-perturb applied to RISC as an additional comparison to confirm this in our setting.

Why not just condition the policy on recent history so it infers the dynamics? The file contains no evidence either way for this in reset-free drift. We therefore run a recurrent-context RISC variant as a baseline. History conditioning adapts the actor, but the critic's targets are still built from stale transitions. It is also complementary to our method, so we test the combination as well.

Proposed Experiments

We take four EARL tasks, Tabletop-Organization, Sawyer-Door, Sawyer-Peg, and Minitaur-Pen, plus Fetch Push-hard from the IBC suite, and add drift schedules in the style of the successor-feature study: step changes, sinusoidal drift, and Ornstein–Uhlenbeck drift. Localized drift changes hinge damping, table and peg friction, or the clearance of the hole. Global drift scales actuator gains or link masses. Following Nguyen and Cheng (provable reset-free RL), who count a manual reset as a constraint violation, the primary metric is cumulative manual interventions over the agent's lifetime. A manual intervention is triggered when the reset policy fails to reach the start distribution within a budget. Secondary metrics are EARL's deployed success rate evaluated under the current dynamics, time to recover to 80% of pre-drift success, and the maximum critic value relative to the discounted return bound, which we use as a divergence indicator. We also report a Sawyer-Peg variant with sparse reward specifically to stress the reward-based detector.

The decisive check is the interaction between drift type and method. The proposal succeeds if validity weighting beats tuned FIFO under localized drift, ties it under global drift, and beats the global-weighting ablation under localized drift. It fails if FIFO wins or ties everywhere, or if the down-weighted transitions do not concentrate in contact phases.