Training a language model with reinforcement learning (RL) is expensive for two separate reasons. The first is generating many sample answers per question. The second is keeping several full-size models in memory at once. Recent work shows that a well-behaved critic can remove the first cost. A critic is a value network that predicts how good a partial answer is. Best Practice Critic Optimization (BPCO) uses bounded value predictions, unnormalized advantages and answer-conditioned critics, and with them it matches a 16-sample Group Relative Policy Optimization (GRPO) baseline while drawing one response per prompt. That critic is a second full model, however, with its own weights, gradients and optimizer state. The obvious fix is to let the policy and the critic share one network. The classic objection, from Phasic Policy Gradient (PPG), is that their gradients interfere when they share weights. PPG's remedy was to go back to separate networks, which roughly doubles memory.
We propose to make the shared network work by controlling how much of each update the value loss is allowed to contribute, layer by layer, while the backward pass is running. One LLM plays both roles: an ordinary forward pass acts as the policy, and a second forward pass with the reference answer in context feeds a bounded value head and acts as BPCO's critic. Each parameter tensor receives its policy gradient and its value gradient in turn. A lightweight hook measures their sizes and their alignment. It rescales the value gradient to a target fraction of the policy gradient, in the spirit of the EnCodec loss balancer, and removes the component of the value gradient that points against the policy gradient. None of this stores a second full-size gradient or optimizer copy. The aim is BPCO's one-sample rollout efficiency at roughly the memory of critic-free training.
Core Claim
Our claim is that interference between policy and value gradients in a shared LLM can be handled by a cheap balancer that streams through the backward pass. Such a balancer needs no phase separation, no second model and no extra full-size buffer. With it, a single-backbone actor-critic should come within noise of a separate-critic BPCO run on math reasoning accuracy, and should cut the peak training memory taken by model state by close to the size of one model's weights, gradients and optimizer state.
The novelty is specific. Adaptive balancers such as GradNorm, MultiAdam and CAGrad have been developed for supervised multi-task vision, physics-informed networks and domain adaptation. GradNorm learns task weights from training rates. MultiAdam keeps separate optimizer moments per loss group. CAGrad projects conflicting detection and alignment gradients. Within the LLM RL literature in the file, the balancing work addresses other pairs of losses. DaGRPO handles conflict among rollouts within a GRPO group, using an LLM judge to mask near-duplicate answers. BRIDGE balances supervised and RL losses through a bilevel teacher with its own auxiliary state. Short-RL gates a length penalty at the reward level. None of these balances the policy loss against a value loss inside one LLM, and none does so as a memory-saving device.
Mechanism
The design has to respect a constraint. MultiAdam's per-group second moments would double optimizer state, which is exactly the memory we are trying to save, so MultiAdam supplies the motivation here but not the implementation. Instead, we use the fact that BPCO's answer-conditioned critic already needs its own forward pass. The policy loss and the value loss therefore backpropagate separately, and each parameter tensor $\theta_\ell$ receives first the policy gradient $g^{p}_\ell$ and then the value gradient $g^{v}_\ell$.
When $g^{v}_\ell$ arrives, a parameter hook performs three steps on that tensor alone:
- It computes the inner product of the two gradients and both of their norms.
- If the two gradients point in opposing directions, it removes the conflicting part of $g^{v}_\ell$. The policy has priority, so only the value gradient is changed.
- It scales the result by a weight $w_\ell$, taken from an exponential moving average of the previous steps' norm ratios. This weight makes the value contribution a target fraction $\rho$ of the policy contribution.
The applied update to tensor $\ell$ is then
$$ \Delta_\ell = g^{p}_\ell + w_\ell\Big(g^{v}_\ell - \min\big(0,\langle g^{v}_\ell, g^{p}_\ell\rangle\big)\,\frac{g^{p}_\ell}{\lVert g^{p}_\ell\rVert^{2}}\Big), \qquad w_\ell = \rho\,\frac{\overline{\lVert g^{p}_\ell\rVert}}{\overline{\lVert g^{v}_\ell\rVert}}, $$
where overbars denote moving averages from past steps, so the weight is known before accumulation. All of this is per-tensor arithmetic on gradients that already exist. The policy gradient for the tensor is simply the contents of its gradient buffer when the value gradient arrives. The overhead should therefore be comparable to GradNorm's reported roughly 5% wall-clock cost on NYUv2 (Chen et al.).
The balancer does keep one hyperparameter, $\rho$. Following the known limitation that automated balancers retain secondary hyperparameters, we treat sensitivity to $\rho$ as a first-class result rather than hiding it.
Research Program
The work proceeds in three steps.
Step 1: measure the interference. Before building anything, we test whether interference actually appears in LLMs. We train a shared policy and critic naively, with summed losses and a fixed value coefficient. We log per-layer cosines between policy and value gradients and the value head's explained variance, and compare them against a separate-critic BPCO run. Two findings would reshape the method. If conflict is concentrated in late layers, the balancer can be restricted to those layers. If conflict is negligible, the proposal collapses to "just share weights," which would itself be a useful negative result.
Step 2: validate on the classic case. We validate the balancer where interference is already established, on the Procgen benchmark, comparing shared-network PPO with balancing against PPG and against shared PPO. This checks that streaming balancing can replace phase separation in the regime where phase separation was invented.
Step 3: run at LLM scale. We scale to LLM math reasoning with a BPCO-style recipe on a 7B-class model, using Qwen2.5-7B, the model used in GHPO. Throughout, we track accuracy, peak GPU memory, wall-clock time per step and rollout tokens consumed.
Baselines
Naive reference points. The first reference point is 16-sample GRPO, the critic-free method that BPCO matches. For it, DaGRPO reports a standard-GRPO average of 44.1% over six math benchmarks: AIME 24/25, AMC, MATH-500, Minerva and OlympiadBench. That 44.1% is derived from DaGRPO's reported 48.8% minus its stated 4.7-point gain. GHPO reports 39.8% average accuracy for Qwen2.5-7B GRPO on its six reasoning benchmarks. These figures come from different setups, so we rerun GRPO ourselves with identical data, prompts and token budget rather than borrowing them.
The second reference point is naive weight sharing: summed policy and value losses with a value coefficient grid-searched over a small set. This "do nothing clever" sharing tells us whether the balancer adds anything.
Strongest existing competitor. The strongest competitor is BPCO with a separate full-size critic. It is the method whose accuracy we must match while saving its critic's memory. No reported numeric accuracy for BPCO was found on Lacuna. The file states only that BPCO matches 16-sample GRPO on AIME 2025 with one response per prompt, so we will reproduce it under our budget. "Beating" BPCO means accuracy within run-to-run noise of separate-critic BPCO, measured over at least three seeds on the six-benchmark average and on the out-of-distribution set (ARC-c, GPQA-Diamond, MMLU-Pro), together with a clearly lower memory peak.
Fair comparison. All arms share the same prompts, the same number of policy updates, the same total generated tokens and the same tuning budget for their single most important hyperparameter: $\rho$ for ours, the value coefficient for naive sharing, and the phase frequency for PPG-style training.
Ablations of the key ingredient. Three ablations isolate what matters in the balancer. The first uses norm-fraction scaling only. The second uses conflict projection only. The third applies both globally rather than per layer.
Why not just…?
Why not just stop the value gradient from reaching the backbone? A value head trained on frozen, detached policy features has zero interference and almost no memory cost, so it is the cheapest possible shared critic. Its weakness is that it can only read features shaped for generating answers. BPCO's evidence is that critics fail for specific trainability reasons, such as unbounded heads and biased bootstrapping, which suggests the critic needs real capacity. We nonetheless treat this baseline as a real threat and run it as a full arm. If its explained variance and downstream accuracy match ours, the conclusion is that sharing is free without balancing, and we will report that.
Why not just give the critic its own low-rank adapter on the shared base? A critic adapter keeps base weights shared and gives the value loss its own small set of parameters. This sidesteps interference with only modest memory cost, which makes it the most serious alternative. It is plausibly as good as our method. The file contains no result on adapter-based LLM critics, so this is settled empirically: we run a critic adapter at several ranks under equal memory. Our method wins only if full-rank balanced sharing yields better value accuracy or reasoning accuracy at equal or lower peak memory. If the adapter wins, the proposal reduces to recommending it.
Why not just stay critic-free with GRPO? GRPO removes the critic but multiplies rollout generation by the group size. Generation is the dominant cost identified in work on natural language policy optimization. BPCO shows that one-sample critic training can match 16-sample GRPO. Our comparison reports accuracy per generated token and per GPU-hour, which directly prices this trade.
Why not just use PPG's phases? PPG avoids interference by keeping separate networks during the policy phase, which is the doubled memory we want to avoid. Its phase frequency is also a sensitive hyperparameter. A PPG-style LLM arm with an equal tuning budget shows whether streaming balancing recovers PPG's stability without its footprint.
Proposed Experiments
Procgen. On Procgen we compare shared PPO, PPG, and shared PPO with our balancer across the standard environments. No single scalar Procgen score for PPG is reported in the file, so the target is to match PPG's normalized score curves under our own reproduction.
LLM math training. For LLM math RL we train Qwen2.5-7B with the six-benchmark math evaluation and the out-of-distribution set above. We report, for every arm: accuracy, peak memory, wall-clock per step, generated tokens, critic explained variance and per-layer gradient cosine.
Decisive checks. Three outcomes settle the proposal. First, the shared-balanced arm must land within noise of separate-critic BPCO while cutting peak memory by most of one model's training state. Second, it must beat naive sharing, which shows that the balancer, not merely sharing, matters. Third, it must match or beat both the detached-head and critic-adapter arms at equal memory. A further check sweeps $\rho$ over an order of magnitude. If accuracy proves fragile to $\rho$, we will report that the method trades one hand-tuned coefficient for another rather than claiming automatic balance.
References
- Best Practice Critic Optimization
- Phasic Policy Gradient
- High Fidelity Neural Audio Compression
- GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networks
- MultiAdam: Parameter-wise Scale-invariant Optimizer for Multiscale Training of Physics-informed Neural Networks
- Conflict-Alleviated Gradient Descent for Adaptive Object Detection
- DaGRPO: Rectifying Gradient Conflict in Reasoning via Distinctiveness-Aware Group Relative Policy Optimization
- Beyond Two-Stage Training: Cooperative SFT and RL for LLM Reasoning
- Efficient RL Training for Reasoning Models via Length-Aware Optimization
- GHPO: Adaptive Guidance for Stable and Efficient LLM Reinforcement Learning
- Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization