Replaying What Is Changing: Boundary-Free Prioritized Replay for Continual Reinforcement Learning

A reinforcement learning agent that learns many skills one after another usually keeps a replay buffer, a memory of past interactions that it re-trains on. It also usually needs someone to announce when one task ends and the next begins. Remove those announcements and the standard way of choosing what to replay breaks down. That standard is prioritized experience replay: sample more often the memories where the agent's value prediction is most wrong, as measured by the temporal-difference (TD) error. The error is huge on every transition from a freshly arrived task, it stays high forever on noisy transitions, and it is largest in absolute terms on whichever task pays the biggest rewards. So a boundary-blind prioritized buffer piles onto the newest or loudest data. It starves consolidation of old skills, or it lets high-reward old tasks drown out a new sparse-reward one. Existing fixes either announce the boundary themselves with a change detector, or give up on prioritization and sample uniformly.

We propose to prioritize a stored transition by how fast its error is changing, not by how large the error is. A transition whose error is rising was previously fit and is now being forgotten, so replaying it consolidates. A transition whose error is falling is being learned right now, so replaying it speeds acquisition. A transition whose error is flat is either mastered or unlearnable noise, and either way replaying it is wasteful. Measuring this change as a scale-free relative quantity has a second effect: high-reward tasks cannot dominate sampling. We claim this single signal gives three things without any task identifiers or change-point detector. It gives task-free consolidation, task-free acquisition, and task-free protection against reward-scale dominance.

Core Claim

Let $\delta_i^{(k)}$ be the TD error of stored transition $i$ at its $k$-th evaluation under the current networks. Transition $i$'s priority is

$$p_i = \epsilon + \frac{\left|\,|\delta_i^{(k)}| - |\delta_i^{(k-1)}|\,\right|}{|\delta_i^{(k)}| + |\delta_i^{(k-1)}| + c},$$

where $\epsilon$ is a small floor that keeps every transition reachable and $c$ is a small constant that avoids division by zero. Sampling probability and importance-sampling correction follow Prioritized Experience Replay Tom Schaul. A new transition has no history yet, so it enters with PER's usual maximal priority until it has been evaluated twice.

Because the priority is a ratio of error magnitudes, multiplying a task's rewards by 100 leaves it unchanged. This is the boundary-free analogue, for sampling, of the per-task value normalization that RECALL needs task IDs to compute.

The central empirical claim is testable. On task-agnostic continual manipulation sequences, a recurrent agent with this "change-prioritized" replay will hold on to old skills better than uniform replay and better than error-magnitude prioritization, and it will not hit the plasticity wall. The gains should be largest exactly where detector-based methods fail: blurry task transitions and sparse rewards.

Where This Sits Relative to Prior Work

Three papers motivate the gap.

Replay-enhanced Continual Reinforcement Learning (RECALL) shows that naive replay fails even with perfect memory. The agent failed to learn the second task in 31 of 100 task pairs, because mastered high-reward tasks dominate learning. RECALL fixes this with per-task value normalization (PopArt) and distillation into frozen per-task policies. Both require task boundaries and task-specific heads, and RECALL samples uniformly.

Task-Agnostic Continual Reinforcement Learning: Gaining Insights and Overcoming Challenges (3RL) shows that a soft actor-critic agent can infer task context from its recent history, using a recurrent GRU history encoder, and can match multi-task performance on Meta-World with no task IDs at all. It also uses uniform replay and reports that the method is computationally demanding. We keep 3RL's architecture unchanged and replace only its sampler, so any gain is attributable to what gets replayed.

Sample Efficient Experience Replay in Non-stationary Environments (DEER) is the closest existing work. It observes, as we do, that TD error conflates policy learning with environment change. It then separates the two with a "discrepancy of environment" score and switches priority rules when a reward-based Jensen-Shannon detector fires. Our proposal differs in mechanism: there is no detector and no discrete switch. Every transition carries its own continuous change signal. It also differs in regime: we target multi-task sequences with blurry boundaries and sparse rewards, whereas DEER studies a single mid-training physics shift and concedes that its detector cannot see shifts that leave rewards unchanged.

Two lines of work constrain the design.

Daochen Zha finds that replaying low-error transitions stabilizes training and prevents forgetting. This rules out any sampler that is monotone in error magnitude. Our signal reconciles that finding with classical PER: low-error transitions are replayed precisely when their error starts to climb.

Uncertainty Prioritized Experience Replay and Reliability-Adjusted Prioritized Experience Replay show that raw error magnitude is corrupted in two ways. Irreducible noise keeps it high, which UPER counters with an epistemic-to-aleatoric ratio from distributional ensembles. Unreliable bootstrap targets inflate it, which ReaPER counters with a trajectory reliability score. Both were studied only in stationary single-task settings. Our signal handles the noise case more cheaply, since high-and-flat error receives the floor priority, but it does not by itself address target unreliability. We therefore test a combination with ReaPER's reliability score.

Prioritized Generative Replay warns that concentrated resampling of few high-priority items drives neurons dormant. We will monitor dormant ratios for the same reason.

Research Program

We will build the sampler on top of the public 3RL setup: soft actor-critic, a GRU history encoder, and trajectory-segment replay. To keep representation drift of stored recurrent states from confounding the comparison, hidden states are always recomputed from stored raw histories with a burn-in prefix, identically for every sampler.

Priorities must be fresh for the change signal to mean anything. PER only re-evaluates transitions when they are sampled, and Distributed Prioritized Experience Replay documents how quickly such priorities go stale. We therefore add a probe refresh: each update also re-evaluates a fixed-size random batch of stored segments. This guarantees every transition a known refresh cadence and bounds the extra cost to a constant number of forward passes per step. To normalize for uneven refresh intervals, the change is divided by the number of gradient steps since the previous evaluation.

The analysis has three parts. First, we run a replay-allocation diagnostic. Task labels are hidden from the agent but used post hoc to measure what fraction of replay goes to tasks whose success is currently dropping, to tasks being learned, and to tasks that are mastered. Second, we split the signal by sign, rising-only versus falling-only, to see which half drives retention and which drives acquisition. Third, we track dormant-neuron ratio and gradient conflict to check that the sampler does not collapse network capacity.

Baselines

The naive reference is 3RL exactly as published, with uniform replay over the same buffer. It is the strongest published task-agnostic continual RL method in the file and isolates the effect of changing only the sampler. We found no reported numeric Continual World success rate for 3RL on Lacuna; the file records only that it reaches multi-task-level performance on Meta-World. We will therefore reproduce it ourselves and treat our reproduction as the reference. Beating it means higher final average success and lower forgetting on CW10, CW20 and MW20, at matched environment steps and matched wall-clock compute, across at least ten seeds with non-overlapping confidence intervals.

The strongest competitor on the non-stationarity question is DEER (Sample Efficient Experience Replay in Non-stationary Environments). It reports an average sample-efficiency improvement of 11.54% over prior replay methods, rising to 22.53% under 200% parameter shifts, on non-stationary MuJoCo. We run DEER inside the same 3RL agent with its detector window and thresholds tuned under the same search budget as our method's $\epsilon$ and refresh size. We also rerun it on its own MuJoCo protocol. Beating it means matching it on abrupt reward-visible shifts, where its detector is at its best, and exceeding it on blurry and sparse-reward streams.

Error-magnitude priority baselines are standard PER, UPER and ReaPER, each dropped into 3RL. They are the natural "just prioritize" competitors and they test whether change beats level.

The task-aware ceiling is RECALL, run with oracle task boundaries. It reports 0.89 average success on CW10 and 0.90 on CW20 in Replay-enhanced Continual Reinforcement Learning. That paper also reports PackNet at 0.87 on CW10, and Continual World: A Robotic Benchmark For Continual Reinforcement Learning reports PackNet at 0.80 on CW20. We do not expect a task-free method to beat an oracle-boundary method. Closing most of the gap between reproduced 3RL and RECALL's 0.89 would count as success.

The key ablation replaces the change signal with the relative level $|\delta|/(|Q|+c)$, using the same normalization, the same probe refresh and the same compute. This separates "change versus level" from "fresher priorities" and "scale-free priorities."

Why not just…?

Why not just keep uniform replay, as 3RL does? It is the real incumbent and may be hard to beat on CW10, where the buffer is short enough that nothing is badly diluted. RECALL's finding that naive replay stalls on 31 of 100 task pairs shows uniform sampling is not safe in general. We still expect the margin to grow with sequence length, so the CW10-to-CW20-to-MW20 progression is where this question is settled. If the gain vanishes on MW20 too, the proposal fails.

Why not just add a global, task-free PopArt normalization to 3RL? This is the most serious threat to the plasticity-wall part of the claim. Normalizing value targets globally may remove reward-scale dominance without any change to sampling. We run 3RL with global PopArt, with our sampler, and with both. If global PopArt alone removes the plasticity wall, we will narrow our claim to retention and sample efficiency rather than argue around it.

Why not just mix high-error and low-error sampling at a fixed ratio? This follows Zha's observation that low-error replay protects old knowledge, and it needs no history per transition. The weakness of a fixed mixture is that it spends consolidation budget uniformly on mastered transitions, whether or not they are being forgotten. We include a tuned fixed-ratio mixture of PER and inverse-PER sampling. The allocation diagnostic will show directly whether change-prioritization concentrates consolidation on the tasks that are actually degrading.

Why not just use DEER's detector with a softer threshold? On blurry streams there is no change point for any threshold to find. In sparse-reward streams the reward distribution does not move at all, as the DEER authors themselves concede. The blurry and sparse variants below test this directly.

Proposed Experiments and Decisive Checks

We evaluate on CW10, CW20 and MW20 with task IDs withheld, following the 3RL protocol. We also build a blurry variant of CW10: consecutive tasks overlap, with the probability of drawing an episode from the next task ramping linearly over a window of steps, and earlier tasks recur at low rates. A sparse-reward variant gives reward only on task success. On the non-stationary MuJoCo protocol of DEER we compare on its home ground. On Continual Bench we give a reference point against the replay-free Online Agent of Continual Reinforcement Learning by Planning with Online World Models, which reports 72.93% average performance against 61.83% for a coreset baseline and under 40% for model-free agents such as SAC. Metrics are average success, forgetting and forward transfer as defined by Continual World, together with wall-clock time and dormant ratio.

Four results would decide the question.

The first is whether change-prioritization beats the relative-level ablation at matched compute. If it does not, the core idea is wrong and fresher, scale-free priorities explain any gain.

The second is whether, on the blurry and sparse variants, it beats DEER by more than on abrupt shifts. If not, the detector-free argument is empty.

The third is whether, in the allocation diagnostic, replay share for a task rises before that task's success rate drops. That would mean the rising-error signal anticipates forgetting rather than trailing it, which is the mechanism we claim.

The fourth is whether the pairwise plasticity-wall failure count falls toward zero without global PopArt, as RECALL achieved with task IDs. If not, the scale-invariance claim reduces to what normalization already provides.