The best current recipe for learning from very little feedback in a changing environment destroys knowledge on purpose. The top sample-efficient Atari agent periodically pulls half of each weight back toward a fresh random initialization. This keeps the network able to learn, a property called plasticity, as its own changing behavior shifts the data it sees. The label-efficient learning community takes the opposite bet. It pretrains an encoder with self-supervision (learning from unlabeled data by solving puzzles built from the data itself) so that a few labels go a long way. These two recipes collide as soon as a pretrained encoder is dropped into a non-stationary setting, meaning one where the data or the targets drift over time. A random reset erases exactly what pretraining bought.
We propose a simple change and a careful test of why it should work. When a pretrained encoder is refreshed, pull its weights back toward the pretrained checkpoint instead of toward random noise. The bet rests on one empirical claim, which we test first: a self-supervised checkpoint is both knowledgeable and still easy to train. If that holds, it is a better restoring point than random initialization on both counts. If it fails, and pretrained checkpoints turn out to be already stiff, the proposal fails cleanly and we learn something the field currently does not know.
Core Claim
Consider an agent or classifier whose encoder starts from self-supervised weights $\theta_{\text{pre}}$ and is then trained on drifting targets. Every $K$ updates, we apply a partial reset:
$$\theta \leftarrow \alpha\,\theta + (1-\alpha)\,(\theta_{\text{pre}} + \sigma\,\epsilon), \qquad \epsilon \sim \mathcal{N}(0, I).$$
Here $\theta$ are the current encoder weights and $\alpha \in [0,1]$ is how much of the current solution survives. The noise scale $\sigma$ is small and keeps the reset from landing on exactly the same point every time. Task heads are still reset toward random initialization, as in BBF. We claim this "reset-to-pretraining" schedule keeps the plasticity benefits of random resets while keeping pretraining's head start. We further claim it beats both random resets and continuous anchoring in low-data drifting regimes.
The claim has a precondition, which we make explicit and test. At the checkpoint, the self-supervised network must have high feature rank (its features spread across many independent directions), few dormant units, and a flat enough loss surface that it fits new targets quickly.
Where This Sits
Three lines of prior work motivate the gap.
Pretraining Representations for Data-Efficient Reinforcement Learning (SGI) pretrains an Atari encoder offline with latent prediction, goal reaching, and inverse dynamics, then fine-tunes it. It reports median human-normalized score 0.753 on Atari 100k. Even with a hand-reduced encoder learning rate, it notes representational collapse in several configurations, and it measures no plasticity diagnostics.
Bigger, Better, Faster (BBF) reaches interquartile-mean human-normalized score 1.045 on Atari 100k from scratch. It relies on periodic 50% perturbations toward random weights, which would erase a pretrained encoder.
Online Pre-Training for Offline-to-Online Reinforcement Learning fixes value collapse when moving from offline to online training by adding a second critic. It leaves the feature encoder's plasticity untouched. Our work changes the encoder side and is orthogonal to that critic design.
The plasticity literature both supplies our tools and constrains our design.
- Understanding and Preventing Capacity Loss in Reinforcement Learning introduces InFeR, which continuously regularizes features toward random-initialization outputs. It also reports that InFeR hurt dense-reward games such as Asteroids and Jamesbond because the anchor was too rigid. This is a warning against continuous anchoring. It is the main reason we reset periodically rather than regularize every step.
- Understanding Plasticity in Neural Networks ties plasticity loss to sharpening curvature and stale optimizer statistics. It also supplies the random-target-fitting probe we reuse to measure plasticity directly.
- Stable Deep Reinforcement Learning via Isotropic Gaussian Representations (SIGReg) prevents rank collapse by shaping features toward an isotropic Gaussian. It is explicitly blind to semantic structure, so it is a competitor rather than a component.
- BarlowRL shows that a redundancy-reduction auxiliary loss lifts DER from 0.183 to 0.278 interquartile mean on Atari 100k. This supports the simpler alternative of "keep a self-supervised loss running."
On the label-efficient side, Self-supervised representation learning from 12-lead ECG data supplies our second testbed. Its predictive-coding encoder on PTB-XL reaches macro AUC 0.927 as a frozen linear probe and 0.9418 when fine-tuned, but only on a static task.
Research Program
The work has three parts.
Part 1: Is a pretrained checkpoint a good restoring point? This is diagnostic and must come first. We take self-supervised checkpoints and random initializations of the same architecture: SGI-style encoders for Atari and the ECG predictive-coding encoder. On each, we measure penultimate feature rank, dormant-unit fraction, and the largest curvature of the loss (top Hessian eigenvalue). We also measure the number of steps needed to fit a fresh random regression target, as in Understanding Plasticity in Neural Networks. We then repeat these measurements along a fine-tuning run on drifting targets. This answers the open question in the file of whether pretraining slows or speeds the decay.
Part 2: The method on Atari 100k. We build reset-to-pretraining on the BBF agent: the same network, the same self-predictive loss, the same reset schedule and update-horizon annealing. The encoder is initialized from SGI-style offline pretraining on the offline trajectory data SGI used. The only change is the encoder reset target. We sweep $\alpha$ and $\sigma$ on a small grid. We also test per-layer $\alpha$, since early convolutional layers are likelier to carry transferable structure.
Part 3: A label-scarce drifting stream built from PTB-XL. This carries the idea outside reinforcement learning. The ECG encoder is fine-tuned on a sequence of PTB-XL label groups: diagnostic superclasses, then subclasses, then rhythm and form statements. Each stage has a small labeled budget, and the sequence is replayed twice so that later stages revisit earlier ones. The drift here is in the target mapping rather than in a policy. This is the case a clinical model faces when label definitions or protocols change. We record macro AUC at each stage and the plasticity probes from Part 1.
Baselines
The naive reference is a pretrained encoder fine-tuned with no resets and a reduced encoder learning rate. This is the SGI protocol, whose best reported figure is median human-normalized score 0.753 (SGI-M/L, Pretraining Representations for Data-Efficient Reinforcement Learning). It tells us whether any reset is needed at all once pretraining is present.
The strongest existing method is BBF trained from scratch, with interquartile-mean human-normalized score 1.045 reported in Bigger, Better, Faster. We rerun it in our codebase rather than quoting the number, so that seeds and evaluation protocol match. Beating it means a higher interquartile mean with non-overlapping stratified bootstrap confidence intervals over the same number of seeds and the same 100k interaction steps. We report pretraining compute separately so readers can judge the offline cost.
The most important competitor is BBF initialized from the same pretrained encoder but keeping random-target resets. This is the obvious "just combine them" system. Together with a variant that resets only the heads and never the encoder, it forms the ablation that isolates our key ingredient. All three share the architecture, the schedule, and an equal hyperparameter grid for reset strength. They differ only in where the encoder is pulled.
Two regularization competitors round out the set, each tuned over the same grid size:
- Continuous anchor: InFeR-style feature regularization toward the pretrained network's outputs, plus a weight-space L2 pull toward $\theta_{\text{pre}}$.
- Geometry regularizer: SIGReg from Stable Deep Reinforcement Learning via Isotropic Gaussian Representations, applied to the pretrained agent.
On the PTB-XL stream, the static fine-tuned figure of 0.9418 macro AUC from Self-supervised representation learning from 12-lead ECG data is a ceiling for a single stage, not a streaming result. No reported result was found on Lacuna for any method on a drifting PTB-XL stream, so all streaming baselines are run by us. Beating them means higher mean stage-wise macro AUC on the second pass, where lost plasticity or forgetting would show.
Why not just…?
Why not just keep BBF's random resets and start from the pretrained encoder? Because a 50% pull toward noise should wipe out most of the pretrained structure within a few resets. The gain would then shrink toward from-scratch BBF. This is the central ablation. If random-target resets on a pretrained start match our method, the claim is wrong and we will say so. A plausible middle outcome is also informative: random resets help late while pretrained resets help early.
Why not just freeze the encoder and train a linear head, or reset only the heads? A frozen encoder cannot follow the state distribution as the policy changes. On static ECG, the frozen probe already trails fine-tuning (0.927 versus 0.9418 macro AUC in Self-supervised representation learning from 12-lead ECG data). SIGReg's analysis also shows that a fixed but badly shaped representation still tracks moving targets poorly. Head-only resets are the fair version of this alternative and are included as an ablation.
Why not just regularize continuously toward the pretrained weights or features? This is the real threat, since it is equally simple. The evidence against it is indirect: InFeR's continuous anchor hurt dense-reward games, which suggests every-step anchoring over-constrains specialization. Periodic resets, by contrast, let the network specialize freely between restorations. We do not assume this carries over to a semantic anchor. The tuned continuous-anchor baseline decides it, with results broken out by dense- versus sparse-reward games.
Why not just keep a self-supervised loss running online, or add LayerNorm? BBF already runs a self-predictive loss and still needs resets. Understanding Plasticity in Neural Networks reports that LayerNorm smooths curvature without fully resolving the stability–plasticity trade-off. Both are therefore kept in every configuration, including ours. They are held fixed, not treated as competitors to beat.
Decisive Checks
The Part 1 probe gates everything. Suppose self-supervised checkpoints fit fresh random targets no faster than random initialization, or show lower feature rank. Then the premise fails, and we report that as the main finding.
The main Atari test is the three-way comparison with everything else fixed. The arms are pretrained start with random resets, pretrained start with pretrained resets, and BBF from scratch. The prediction is that our method wins most clearly in the first half of the 100k steps, where pretraining matters most. It should also keep interquartile mean at or above from-scratch BBF by the end.
Mechanism check. Across all arms, we measure feature rank and dormant fraction right after each reset. We also measure the self-predictive loss on held-out offline trajectories as a readout of retained pretrained knowledge. The method should restore rank as random resets do, while keeping the held-out loss near its pretrained value.
Generality check. On the PTB-XL stream, the method should raise second-pass macro AUC relative to fine-tuning without resets. This would show the benefit is not specific to policy-induced drift.
References
- Pretraining Representations for Data-Efficient Reinforcement Learning
- Bigger, Better, Faster: Human-level Atari with human-level efficiency
- Online Pre-Training for Offline-to-Online Reinforcement Learning
- Understanding and Preventing Capacity Loss in Reinforcement Learning
- Understanding Plasticity in Neural Networks
- Stable Deep Reinforcement Learning via Isotropic Gaussian Representations
- BarlowRL: Barlow Twins for Data-Efficient Reinforcement Learning
- Self-supervised representation learning from 12-lead ECG data