Engineers who write reward functions for robots rarely get the weights right on the first try. A common repair is to change the weights during training. One version starts with a tiny penalty for falling and ramps it up. Another trains on the easy "make progress" terms first and switches on the constraint penalties later. These schedules often work at first and then break. Right after the switch, performance collapses, and sometimes it never recovers. This note proposes that the collapse is mainly a form of primacy bias, the tendency of a deep reinforcement learning (RL) agent to overfit its networks to whatever it saw early in training and lose the ability to learn from new signals, often called loss of plasticity. The critic is the network that predicts future reward. It spent the first phase fitting one weighting of the reward terms, and it cannot bend to fit the second.
The remedy we test is structural and cheap. Give the critic one output head per reward term, and train every head from the first step, including heads whose term currently carries a weight near zero. The agent's value under any weighting is then a weighted sum of heads that already exist. Changing the reward becomes a recombination of existing predictions instead of a new regression problem. If this is right, a practitioner can reshape a composite reward mid-training without triggering the entrenchment that currently forces full network resets. We also expect the remaining failures to localize to a small, measurable set of heads and shared features, which makes targeted plasticity repairs possible.
Core Claim
Write the engineered reward as $r = \sum_{i=1}^{m} w_i r_i$, where the $r_i$ are the individual terms (forward velocity, torque penalty, fall penalty, stage bonus) and $w_i$ are the weights the engineer chooses. A decomposed critic learns one value $Q_i(s,a)$ per term, the expected discounted sum of $r_i$ under the current policy. It then acts on
$$Q_w(s,a) = \sum_{i=1}^{m} w_i\, Q_i(s,a).$$
This identity holds exactly for policy evaluation because the expected return is linear in the reward. Soft actor-critic's entropy bonus is added to the reward inside the value, so we treat it as one more head.
Our claims are empirical:
- With a single scalar critic, the deterioration seen at reward-schedule switches is a plasticity failure, not a failure of the replay data.
- With per-term heads trained from step one, that deterioration largely disappears for any schedule, however abrupt.
- The residual failures concentrate in shared-trunk features. Those are features that dense early terms shaped and that carry little information about rare terms, such as goal arrival or contact. Fixing them requires plasticity injection only there.
Where This Sits
Three papers motivate the proposal by documenting the failure without naming its mechanism:
- Curriculum Reinforcement Learning for Complex Reward Functions trains first on a simplified reward subset and then switches to the full constrained reward. It reports severe policy deterioration at the switch, which it leaves unexplained.
- Value Function Decomposition for Iterative Design of Reinforcement Learning Agents builds SAC-D, a critic with one head per reward term. Its per-term influence metric shows that the failure penalty paralyzes early learning on Bipedal Walker Hardcore. It fixes this by manually ramping the penalty weight from 0.01 to 1.0. SAC-D uses the heads as a diagnostic for a human designer; it does not use them to make weight changes safe, and it does not study plasticity.
- ARS: Adaptive Reward Scaling for Multi-Task Reinforcement Learning is the closest existing work. It shows that rescaling rewards mid-training disrupts the critic and that, without periodic full resets, primacy bias blocks adaptation. It addresses per-task scale in multi-task Meta-World, not competing terms inside one task, and it pays for adaptation with full resets.
Our proposal differs from ARS in mechanism. ARS destroys and relearns the critic so it can adapt to new scales. We try to make the critic's knowledge invariant to the weights in the first place.
Other work supplies methods and constraints:
- The Primacy Bias in Deep Reinforcement Learning gives the diagnostic we reuse. A stalled agent's replay buffer, handed to a freshly initialized network, yields fast recovery. That shows the fault lies in the parameters, not the data.
- A Forget-and-Grow Strategy for Deep Reinforcement Learning Scaling in Continuous Control warns that resets fail when the buffer is dominated by early transitions. We take this seriously. Storing raw per-term rewards lets every old transition be relabeled under the new weights, so stale reward labels cannot contaminate the buffer. Stale state distributions still can, and we test for that.
- Fisher-Guided Selective Forgetting injects noise scaled by the inverse of the empirical Fisher information, a per-parameter measure of how strongly the loss depends on each weight. It targets only the parameters that have lost plasticity and avoids the performance dips of full resets. We reuse it, restricted to the trunk features implicated in the residual failures.
- Confronting Reward Overoptimization for Diffusion Models found that in diffusion fine-tuning, resetting active critic neurons reduces reward hacking, while resetting dormant ones hurts. This constrains our choice of which units to perturb.
- Scaling Off-Policy Reinforcement Learning with Batch and Weight Normalization shows that normalization alone, in the CrossQ + WN architecture, preserves plasticity at high update ratios. That makes it a serious alternative explanation for any gains we see.
Research Program
The work has three parts.
Diagnosis. We establish that switch-time collapse in scalar critics is primacy bias. We train scalar-critic soft actor-critic under a two-phase or ramped reward schedule. At the switch, we branch the run three ways:
- continue as is;
- relabel the buffer with the new weights and continue;
- copy the relabeled buffer into a freshly initialized agent.
If the fresh agent recovers much faster than the continued one, the collapse is a plasticity failure. Alongside the branches we track the Fisher trace, the fraction of dormant units, and the effective learning rate.
The weight-invariant critic. We build a critic with a shared trunk and $m$ heads, trained on relabeled per-term rewards from the first step. The actor is updated on $Q_w$ under the current schedule. Heads for terms with near-zero weight still learn, because their rewards are observed on every transition regardless of weight.
At a switch, the critic needs no refitting. The heads must still track the policy as it adapts, but that is ordinary evaluation drift, not relearning. The actor, however, was optimized for the old weighting. We therefore measure actor plasticity separately to see whether the actor, not the critic, becomes the bottleneck.
Residual repair. Some terms will be nearly invisible early, for example a goal bonus the agent has never reached. For these we expect the trunk to lack features even though the head is correct in form. We test this in two ways:
- A frozen-trunk probe: freeze the trunk and train a fresh linear head on late-phase data for that term. Poor probe fit means the features are missing.
- A targeted repair: Fisher-guided noise applied only to trunk units whose gradients come mostly from the dominant early heads, compared against fully separate per-head trunks as an upper bound.
Baselines
The naive reference is scalar-critic soft actor-critic trained on the full composite reward with final weights from step one. It shows how much any schedule helps at all.
The second reference is the manual schedule applied to a scalar critic, in two forms: the gradual 0.01-to-1.0 penalty ramp of SAC-D's paper, and the abrupt two-phase switch of the reward-curriculum paper. Both use buffer relabeling so that stale labels are not the confound. This is what practitioners do today.
The strongest existing plasticity methods, each applied to the scalar-critic schedule, are:
- full resets at the switch, as in ARS and the primacy-bias paper;
- FoG;
- CrossQ + WN.
For reference, the file reports the following:
- FoG reaches "a total normalized score of 0.92" across 41 continuous-control tasks and "a 53% improvement over standard SAC with resets" on humanoid-walk.
- FGSF reports "a 25% improvement over state-of-the-art Reset methods" on DMC Humanoid.
- On multi-stage Meta-World, SDME (State-Dependent Maximum Entropy RL) reports "an 86% success rate, nearly double the 44% of the best SAC baseline."
The file contains no reported numbers on Lacuna for the constrained DMC variants of the curriculum paper or for final returns on Bipedal Walker Hardcore under SAC-D. On those tasks every baseline is rerun by us.
SAC-D itself, with its conflict-averse gradient combination, is also run under the same schedules.
Ablations isolate the key ingredient:
- decomposed heads trained only once their weight turns on, which tests whether "train every head from step one" matters;
- decomposed heads with the trunk frozen after phase one;
- the proposed method with and without targeted Fisher noise.
All methods share network width, update-to-data ratio, environment-step budget, and a matched tuning budget. The budget is a fixed number of configurations per method on a held-out task. Results use at least ten seeds with interquartile-mean aggregation.
The method counts as beating a baseline if two conditions hold. First, final return under the full reward is at least as high. Second, it significantly reduces the switch-time dip, measured as the maximum drop in normalized return within a fixed window after the switch, and the recovery time. For multi-stage Meta-World we additionally aim to match or exceed SDME's reported 86% without SDME's state-class entropy rules.
Why not just…?
Why not just ramp the penalty gradually, as SAC-D did? This is the real threat. A slow enough ramp may keep a scalar critic close enough to its targets that it never entrenches. We test ramp speed directly, from abrupt to very slow. The proposal is only worth adopting if the decomposed critic stays flat across ramp speeds while the scalar critic degrades as ramps get faster. If slow ramps match us everywhere, the honest conclusion is that the contribution shrinks to "you no longer need to tune the ramp," and we will report it that way. One point cuts the other way: a schedule is itself manual reward engineering, and it requires knowing which terms to delay.
Why not just reset the networks at each switch? Resets throw away the phase-one policy, which produces exactly the performance dip we aim to remove. FoG also argues that a reset network relearns biases from the buffer. The reset-at-switch baseline measures this cost directly on both metrics.
Why not just use a plasticity-preserving architecture like CrossQ + WN? That paper shows normalization keeps learning rates and units alive. It does not remove the need to refit a scalar value when the target function itself changes. We run CrossQ + WN both with a scalar critic and with our heads. If the scalar version closes the gap, normalization is the operative fix. If the two combine additively, the mechanisms are distinct.
Why not just relabel the replay buffer with the new weights and keep training? Relabeling is included in every scalar baseline. The transplant diagnostic asks the precise question here: when a fresh network trained on the same relabeled buffer recovers faster than the continued one, the data are fine and the parameters are the problem.
Proposed Experiments
The decisive checks run on three settings. The main one is DMC locomotion (cheetah, walker, humanoid) with added torque, jerk, and fall penalties, mirroring the curriculum paper's constrained variants. The second is Bipedal Walker Hardcore, the task where SAC-D's penalty freeze was documented. The third is Meta-World drawer-put-in and box-pick-out, whose stage milestones we decompose into one head per stage bonus.
The first test is the transplant diagnostic on scalar critics. Its outcome decides whether the premise holds; if fresh networks do not recover faster, the plasticity story is wrong and we stop there.
The second test is the switch-dip comparison across schedule speeds, run against every baseline above.
The third test is the frozen-trunk probe for rare-signal heads. It decides whether targeted Fisher noise is needed, or whether per-term heads alone suffice.
References
- Curriculum Reinforcement Learning for Complex Reward Functions
- Value Function Decomposition for Iterative Design of Reinforcement Learning Agents
- ARS: Adaptive Reward Scaling for Multi-Task Reinforcement Learning
- The Primacy Bias in Deep Reinforcement Learning
- A Forget-and-Grow Strategy for Deep Reinforcement Learning Scaling in Continuous Control
- Fisher-Guided Selective Forgetting: Mitigating The Primacy Bias in Deep Reinforcement Learning
- Confronting Reward Overoptimization for Diffusion Models: A Perspective of Inductive and Primacy Biases
- Scaling Off-Policy Reinforcement Learning with Batch and Weight Normalization
- State-Dependent Maximum Entropy Reinforcement Learning for Robot Long-Horizon Task Learning