Model-free reinforcement learning agents can learn from far fewer environment interactions if they take many gradient steps on each collected transition, a quantity called the replay ratio. In practice, pushing the replay ratio high makes networks lose their ability to keep learning, and the only reliable fix so far is to periodically wipe the network weights and retrain from the stored data. This proposal argues that the reset is a crude stand-in for something the optimizer already offers in continuous form. AdamW's weight decay, the term that shrinks every weight toward zero each step, makes the weights an exponential moving average of past updates, so it sets a "forgetting clock." We propose a single scaling rule that fixes how fast the network forgets, measured in environment steps rather than gradient steps, and that holds this clock constant as both network width and replay ratio grow. Learning rate and weight decay then change in a predictable way with both. If the rule is right, hyperparameters tuned once on a small, low-replay-ratio agent should transfer unchanged to a wide, high-replay-ratio agent, with no resets.
The idea combines two results that have not been joined. Work on width-dependent parameterization shows how learning rates must shrink with width so that features keep changing at the same speed in narrow and wide networks. Work on sample-efficient RL shows that high replay ratios pay off only when plasticity, the network's ability to absorb new data, is protected. Width scaling has been brought to RL, and replay-ratio scaling has been made to work with resets. Nobody has treated the replay ratio as a scaling axis in its own right with its own rule, and nobody has asked whether a correctly set weight-decay timescale can replace resets.
Core Claim
Write AdamW's update as $w_t = (1-\eta\lambda)\,w_{t-1} - \eta\,\Delta_t$, where $\eta$ is the learning rate, $\lambda$ the weight decay and $\Delta_t$ the normalized Adam step. How to set AdamW's weight decay as you scale model and dataset size shows that this makes the weights a moving average with memory $\tau_{iter} = 1/(\eta\lambda)$ gradient steps. It also proves that keeping $\tau$ fixed keeps training trajectories the same in scale-invariant networks. Under maximal update parameterization (μP, the rule that scales hidden-layer learning rates as $1/n$ in width $n$ so features keep learning in the wide limit), this requires $\lambda \propto n$.
In RL with replay ratio $\rho$ (gradient updates per environment step), the relevant memory is in environment steps, because the data distribution shifts as the policy improves:
$$\tau_{env} = \frac{1}{\rho\,\eta\,\lambda}.$$
Our central hypothesis is that the optimal $\tau_{env}$ is roughly invariant across width and replay ratio. It is set by how fast the policy and the buffer turn over, not by the network.
Holding $\tau_{env}$ fixed constrains only the product $\eta\lambda$, so a second rule is needed to split it. We borrow the duration rule from Completed Hyperparameter Transfer across Modules, Width, Depth, Batch and Duration, which derives $\eta \propto 1/\sqrt{\kappa}$ for training duration $\kappa$. At a fixed budget of environment steps, the number of gradient steps grows in proportion to $\rho$. That gives the joint prediction
$$\eta \propto \frac{1}{n\sqrt{\rho}}, \qquad \lambda \propto \frac{n}{\sqrt{\rho}} \quad \text{(hidden layers)},$$
with embedding and readout layers following their μP exponents.
The duration rule was derived for static data. Whether it survives replay, where the same transitions are revisited under a moving distribution, is the riskiest part of the claim, and it is tested directly.
A further consequence is that continuous decay at the right $\tau_{env}$ should do the job that resets do, forgetting brittle early features, without throwing away everything at once.
Relation to Prior Work
The closest work is CompleteP for RL. It applies μP plus $1/L$ residual depth scaling to PPO and SAC with MLP residual networks, and shows learning-rate transfer from width 32 to 2048, fewer dormant neurons, and 31% less compute than NTK parameterization on Humanoid Maze. It supplies our base parameterization. However, it holds the replay ratio and buffer fixed and does not scale weight decay, so it says nothing about the axis we target.
Sample-Efficient Reinforcement Learning by Breaking the Replay Ratio Barrier motivates the proposal. It shows that replay ratio 128 trades compute for data: 200k samples match 1M samples on the DeepMind Control Suite. That only works with hand-scheduled full resets every 2.56M updates in standard parameterization. We ask whether those resets are compensating for a mis-set forgetting timescale.
The AdamW timescale paper and the Complete(d)P paper supply the scaling machinery, but both are verified only on static supervised datasets. Our rule is their extension to a sliding, non-stationary buffer.
CrossQ acts as a constraint and a competitor. It reaches the sample efficiency of replay ratio 20 at replay ratio 1 by widening critics to 2048 and using batch normalization without target networks. The Effect of Network Width on Stochastic Gradient Descent and Generalization and the AdamW timescale paper both note that batch normalization complicates width-scaling rules, so we keep our method normalization-free and compare to CrossQ rather than merge with it.
The finding of Etai Littwin et al. that Adam's $\epsilon$ must scale with width is a confound. We remove it by using Adam-atan2 from Katie E Everett et al., which drops $\epsilon$ entirely, with weight decay decoupled as in AdamW.
Research Program
We build SAC with CompleteP-parameterized MLP residual actor and critic, trained with Adam-atan2 and decoupled weight decay set by the joint rule above. The work has three parts.
First, we measure the empirical optimum. On a width grid from 32 to 2048 and replay ratios 1, 4, 16, 32 and 128, we sweep $\eta$ and $\lambda$ on a subset of DeepMind Control tasks (Yuval Tassa et al.). We then check two things: whether optimal $\rho\eta\lambda$ is flat across the grid, and whether the split between $\eta$ and $\lambda$ follows the $1/\sqrt{\rho}$ prediction.
Second, we test zero-shot transfer. We tune once at width 64 and $\rho = 1$, then deploy at width 2048 with $\rho$ up to 128 without further tuning and without resets.
Third, we run mechanistic diagnostics:
- the dormant-neuron fraction;
- a plasticity probe that refits the critic to fresh random targets at checkpoints;
- weight norm, which the AdamW timescale paper predicts scales as $1/\lambda$;
- Q-value overestimation.
These tell us whether the rule preserves plasticity directly, or merely avoids collapse by learning slowly.
Baselines
The naive reference is SAC in standard parameterization with its default fixed learning rate, no weight decay, and the replay ratio simply raised. Breaking the Replay Ratio Barrier reports this collapses, and plain SAC reaches an interquartile-mean (IQM) return of 391 on DMC15. We run it at every grid point to show the failure the rule must fix.
The strongest competitor is SR-SAC, which uses periodic full resets at replay ratio 128 and reports an IQM of 740 on DMC15 (REDQ: 511), all from that paper. It is the strongest because it is the established way to reach high replay ratios and it is algorithm-agnostic. We will cite the published 740 and also reproduce SR-SAC on our exact residual MLP architecture, environment-step budget and seeds. Its reset interval gets the same tuning budget, spent at the same proxy scale, that our rule gets for $\eta$ and $\lambda$.
Beating SR-SAC means two things. First, our transferred, reset-free agent must match or exceed its IQM at 200k and 500k environment steps with overlapping-or-better stratified bootstrap intervals. Second, it must do so with fewer total gradient FLOPs, since resets force relearning.
CompleteP for RL with fixed replay ratio and its own weight decay settings is the third baseline. It isolates whether scaling with $\rho$ adds anything beyond width scaling. No per-task returns for it are reported in the file ("no reported result found on Lacuna" for its DMC scores).
CrossQ at replay ratio 1 with 2048-unit critics is run on MuJoCo Humanoid, Ant, Walker2d and Hopper. The file reports that it matched REDQ and DroQ at replay ratio 20 with about 4× wall-clock speedup, but gives no return values ("no reported result found on Lacuna for CrossQ per-task returns"). We compare on both sample efficiency and wall-clock time.
Three ablations isolate the key ingredient, the forgetting clock:
- the full rule with $\lambda$ held fixed across width, which the AdamW timescale paper predicts breaks transfer;
- the rule with $\eta \propto 1/\sqrt{\rho}$ but $\lambda$ fixed across $\rho$, so lower learning rate without a constant forgetting timescale;
- the full rule plus SR-SAC-style resets, to see whether resets still add anything.
Why not just…?
Why not just keep resetting? Resets work, and SR-SAC's 740 is the bar. However, their interval is hand-tuned per algorithm, they discard all learned features, and they already fail for convolutional encoders, which needed a separate shrink-and-perturb fix in the same paper. The reset-plus-rule ablation settles this. If adding resets to our rule still gives a clear gain at $\rho = 128$, then continuous forgetting does not substitute for resets, and we will report that rather than claim reset-free training.
Why not just lower the learning rate as the replay ratio rises, and tune it per setting? This is the most serious threat. A per-configuration tuned learning rate is a legitimate oracle, and if it alone recovers SR-SAC-level performance, the weight-decay component is unnecessary. Our grid includes exactly this oracle, with $\eta$ tuned at every $(n, \rho)$ and $\lambda = 0$. The proposal's value then rests on two separate claims. One is transfer: the rule matches the oracle without per-setting tuning. The other is necessity: the fixed-$\lambda$ ablation falls short of the oracle-with-decay at high $\rho$. If both fail, the forgetting-clock hypothesis is wrong.
Why not just use CrossQ's wide batch-normalized critics at replay ratio 1? CrossQ attacks sample efficiency on a different axis, cheaper updates rather than more of them, and it may well win on wall-clock time. But it uses standard parameterization at a single width, and batch normalization is known to break the scaling rules we rely on. We therefore treat it as a head-to-head baseline on MuJoCo rather than a component. If CrossQ dominates our method in both samples and wall-clock time, the practical case for high replay ratios weakens, and we would say so.
Why not just add shrink-and-perturb or plain L2 regularization? These are also continuous forgetting, but their strength is not tied to width or replay ratio. We include shrink-and-perturb at a single tuned strength, transferred across the grid without retuning. If it transfers as well as our rule, the scaling exponents carry no information.
Decisive Checks
The proposal stands or falls on four observations on DMC15 and the MuJoCo locomotion tasks, each with at least ten seeds and IQM with bootstrap intervals:
- The optimal $\rho\eta\lambda$ stays within a narrow band across width 32 to 2048 and replay ratio 1 to 128. If instead it drifts systematically, the core invariance is false.
- The $\eta$ versus $\lambda$ split tracks $1/\sqrt{\rho}$. If it fails here but check 1 holds, only the duration-rule borrowing is wrong, and we would fit the exponent empirically.
- Hyperparameters tuned at width 64 and $\rho = 1$ transfer to width 2048 and $\rho = 128$ and match or beat our reproduced SR-SAC, with the published 740 as a reference, using no resets and fewer gradient FLOPs.
- Dormant-neuron fraction and plasticity-probe error stay flat over training under the rule, while they rise under the fixed-$\lambda$ ablation. This would show that the constant forgetting timescale, not merely a smaller step size, is what keeps the agent learning.
References
- How to set AdamW's weight decay as you scale model and dataset size
- Completed Hyperparameter Transfer across Modules, Width, Depth, Batch and Duration
- CompleteP for RL: Maintaining Feature Learning When Scaling Deep Reinforcement Learning
- Sample-Efficient Reinforcement Learning by Breaking the Replay Ratio Barrier
- CrossQ: Batch Normalization in Deep Reinforcement Learning for Greater Sample Efficiency and Simplicity
- The Effect of Network Width on Stochastic Gradient Descent and Generalization: an Empirical Study
- Adaptive Optimization in the $\infty$-Width Limit
- Scaling Exponents Across Parameterizations and Optimizers
- DeepMind Control Suite