Suppose we want a language model to do well on a check that is expensive to run, such as executing code against unit tests, asking a human, or querying a large judge model. A common workaround is to train a cheap learned reward model and optimize the language model against that instead. The usual way to train the reward model is to make its scores match the expensive check on the samples we can afford to label. That is not quite the goal. The goal is a reward model such that the policy trained on it scores well on the expensive check. Framing it this way makes reward design a bi-level problem: an inner loop trains the policy on the learned reward, and an outer loop adjusts the reward so the trained policy does well on the true criterion. This framing is attractive, but in practice it has been too costly for large models. The exact gradient of the outer loop runs through the inner training, and computing it requires inverting the Hessian (the matrix of second derivatives) of the policy's loss over billions of weights.
This note argues that, in the regime language-model RL actually uses, most of that cost is self-inflicted. It comes from writing the inner problem in terms of network weights. When the policy is trained to maximize reward minus a KL penalty to a reference model, the best possible policy has a closed form. Differentiating through that closed form gives the outer gradient as a simple covariance between the expensive check and the reward model's gradient. This quantity can be estimated from the rollouts the policy is already generating, reweighted by importance weights built from log-probabilities that PPO- or GRPO-style training already computes. The result is a bi-level reward update with no Hessian, no differentiation through the policy update, and no extra generation. That lets us ask a question the literature has not settled at scale: at equal cost, does decision-aware reward learning beat simply fitting the reward to the labels?
Core Claim
Let $\pi_{ref}$ be the frozen reference model, $r_\phi(x,y)$ the learned reward model with parameters $\phi$ scoring response $y$ to prompt $x$, and $\beta$ the KL penalty weight. The inner problem, maximizing expected $r_\phi$ minus $\beta$ times KL to $\pi_{ref}$ over all distributions on responses, is solved by
$$\pi^*_\phi(y\mid x) = \frac{\pi_{ref}(y\mid x)\, \exp(r_\phi(x,y)/\beta)}{Z_\phi(x)},$$
where $Z_\phi(x)$ normalizes over responses. The same closed form underlies direct preference optimization. Let $G(x,y)$ be the expensive outer criterion. Differentiating the outer objective $\mathbb{E}_{y\sim\pi^*_\phi}[G(x,y)]$ gives
$$\nabla_\phi\, \mathbb{E}_{\pi^*_\phi}[G] = \frac{1}{\beta}\, \mathrm{Cov}_{y\sim\pi^*_\phi}\big(G(x,y),\ \nabla_\phi r_\phi(x,y)\big).$$
The implicit-function-theorem inverse Hessian has not been approximated away here. In distribution space, the inner objective's Hessian is the Hessian of the KL term, which is diagonal with entries $\beta/\pi$. Inverting it is trivial, and the covariance is exactly what remains. The price is that we must sample from $\pi^*_\phi$, which we cannot do directly.
The practical estimator draws $n$ responses $y_i$ from the current policy $\pi_\theta$, which is being trained toward $\pi^*_\phi$. It then uses self-normalized importance weights $w_i \propto \pi_{ref}(y_i)\exp(r_\phi(y_i)/\beta)/\pi_\theta(y_i)$ to form a weighted covariance over the subset of responses on which $G$ was actually evaluated. Every quantity in $w_i$ is already computed in a KL-regularized RL step, since the reference log-probabilities are needed for the KL penalty. The only new backpropagation is through the reward model, which is small.
The claim has two parts. The first is computational: the reward update adds near-zero wall-clock and memory overhead to an ordinary GRPO/PPO step, which does not hold for any existing bi-level RL method we found. The second is scientific: this decision-aware update gives better final policies than prediction-focused reward fitting at matched expensive-label budgets, especially when the reward model is small relative to the policy.
Relation to Prior Work
Several papers motivate the target but leave cost unsolved. PARL casts RLHF as bi-level and shows that the outer objective must account for how the policy's samples shift. Our covariance under $\pi^*_\phi$ captures exactly that dependence. However, PARL names Hessian computation as its scaling barrier and is tested only on control tasks. AIHF jointly learns reward and policy for a 7B model by alternating full PPO phases with reward updates, and reports that this is substantially more expensive than the sequential pipeline. We keep the joint objective but remove the alternation cost. Principal-driven reward design derives hypergradients via the implicit function theorem, but only in GridWorld.
The closest methods attack the Hessian directly. SoBiRL differentiates the soft Bellman fixed point to obtain a first-order hypergradient with a log-probability trick. It is our nearest relative, and it supplies the insight that entropy regularization makes the solution map tractable. Its theory is tabular, it is evaluated on Atari and MuJoCo, and it requires fresh environment rollouts for the hypergradient. We specialize to the sequence-level setting, where the fixed point is fully closed-form, and we reuse the policy's own rollouts through importance weighting.
The Hessian-free actor-critic uses a penalty reformulation with decaying entropy and multi-time-scale step sizes. It is tested on GridWorld and a 4-layer transformer, and its step-size schedules are delicate. Sample-complexity bounds for bilevel RL proves rates under a Polyak-Łojasiewicz assumption, a condition that bounds how flat the loss can be away from its minimum. However, its authors had to swap in a heuristic upper bound in their experiments. This is a constraint that argues for estimators that are directly implementable.
On the supervised side, ScaleBiO shows that first-order minimax reformulations scale bi-level data reweighting to 32B models. Its formulation relies on differentiable supervised losses and does not cover policy rollouts. Meta-Reward-Net and LIRPG truncate to one inner step, which is cheap but myopic. They still need vector-Jacobian products through the policy or Q-function update.
On the cost side, RL4LMs/NLPO and P-TA document the generation-in-the-loop and multi-model memory costs we refuse to increase. Execution-based code RL documents the external-verifier latency that makes a limited $G$ budget realistic.
Research Program
The method has three components, each tested separately.
The first is the estimator. The policy is trained by GRPO on $r_\phi$ with a KL penalty. Each batch's rollouts are scored by $r_\phi$, and a random fraction $\rho$ of them is sent to the expensive check $G$. The reward model then takes one step along the importance-weighted covariance. Estimator variance is monitored through the effective sample size of the weights. Because $\pi_\theta$ is trained toward $\pi^*_\phi$, the weights should stay moderate. Where they do not, the estimator becomes unreliable, and we will measure how often that happens.
The second is an analysis of bias. The estimator is exact only if the importance weights are correct, and it assumes the policy class can represent $\pi^*_\phi$. We will bound the bias of the self-normalized estimator in terms of the chi-square divergence between $\pi_\theta$ and $\pi^*_\phi$. We will check empirically whether the reward update should be slowed when that divergence is large, which gives a principled two-time-scale rule in place of hand-tuned schedules.
The third is the comparison that matters scientifically. Fitting $r_\phi$ to $G$ by regression has gradient $\mathbb{E}[(G - r_\phi)\nabla_\phi r_\phi]$ under whatever distribution the labels come from. The covariance update instead raises $r_\phi$ wherever $G$ is above average under the target policy, whether or not $r_\phi$ already equals $G$ there. The two coincide when the reward model can represent $G$ exactly and the labels are on-policy. They diverge when a small reward model must choose which distinctions to get right. We will control this by varying the reward model's size.
Baselines
The naive reference is GRPO trained directly on $G$, using only the $\rho$ fraction of rollouts that receive a label. This is what a practitioner does without a learned reward, and it uses exactly the same verifier budget. A second naive reference is the frozen reward model trained once before RL, the standard sequential pipeline.
The strongest competitor on the scientific question is online reward fitting. The same reward model is updated by regression or a Bradley-Terry loss on the same labeled on-policy rollouts at every step, in the style of PEBBLE's interactive reward learning (PEBBLE). It costs the same as our method and differs only in the reward update direction. That makes it the fairest test of whether bi-level reasoning adds anything.
The strongest existing bi-level methods are AIHF and the Hessian-free actor-critic. On the Open LLM Leaderboard, AIHF reports a 64.75% average for Mistral-7B, against 61.68% for the Zephyr base, and 72.63% RewardBench accuracy for its reward model. IRFT reports Zephyr-7B improving from 59.47% to 61.03%. For the Hessian-free actor-critic, SoBiRL, and GRPO on GSM8K under a verifier budget, no reported result found on Lacuna. All of these will be rerun on our base models.
Fairness is enforced in four ways. Every method gets the same base model, prompts, number of generated tokens, and number of $G$ evaluations. Each method gets the same hyperparameter search budget (a fixed number of trials over learning rates, $\beta$, and time-scale ratios). Wall-clock time and peak memory are reported on identical hardware. AIHF is run with its PPO phases counted against the same generation budget.
Two ablations isolate the key ingredients. The first drops the importance weights, treating $\pi_\theta$ as if it were $\pi^*_\phi$, to test whether the correction toward the closed-form optimum matters. The second replaces the covariance with the regression gradient while keeping everything else fixed; this is the online-fitting baseline viewed as an ablation.
Beating the field means three things at matched $G$ budget and within roughly 10% of plain GRPO's wall-clock:
- a higher final score on the held-out outer criterion than online reward fitting;
- matching or exceeding AIHF's quality at a fraction of its compute;
- per-step overhead that the actor-critic and one-step meta-gradient methods cannot match at 7B scale.
Why not just…?
Why not just fit the reward model to the labels online? This is the real threat, and the proposal is built around testing it. If the reward model is large enough to represent $G$ and the labels are drawn on-policy, the two updates nearly agree, and we expect no gain. We predict an advantage only when the reward model is small or misspecified, because then it matters which errors it makes. The reward-model-size sweep settles this. If online fitting matches us at every size, we will report that the bi-level framing buys nothing beyond its computational equivalence, which is itself a useful negative result.
Why not just run RL on the expensive check directly? When $G$ is cheap, this is the right answer, and our GSM8K setting deliberately simulates an expensive verifier by limiting $\rho$. The question is whether a learned reward that generalizes to the unlabeled $1-\rho$ of rollouts extracts more from a fixed verifier budget. Sweeping $\rho$ from small values to 1 will show the crossover point, and we expect our advantage to vanish at $\rho = 1$.
Why not just use DPO or implicit reward fine-tuning? DPO uses the same closed-form optimum but fixes the reward to be the policy's own log-ratio, and it trains offline. AIHF reports that the DPO-implicit reward reaches only 51.21% on RewardBench versus 72.63% for its jointly learned reward. PARL argues that ignoring the policy's distribution shift causes degradation. IRFT is included as a baseline so that the comparison is measured on our setting rather than assumed.
Why not just truncate to a one-step meta-gradient, or scale up the penalty actor-critic? The one-step approach in Meta-Reward-Net and LIRPG still differentiates through a policy update, which is prohibitive in memory at 7B, and it is myopic. In a supervised setting, the penalty method beat one-step ApproxGrad, 79.03% versus 68.08% on noisy CIFAR-10. The penalty actor-critic needs a delicate multi-time-scale schedule. We run both at small scale, where they fit, and compare hypergradient fidelity directly.
Proposed Experiments
The first experiment checks hypergradient fidelity at small scale, before any large runs. We use the Happy Tweet setup from the Hessian-free actor-critic paper and IMDB sentiment from the GRUE benchmark, with small policies where long unrolled differentiation is feasible. We compute a reference hypergradient by long unrolls and finite differences. We then report the cosine similarity to that reference for our estimator (with and without importance weights), the one-step meta-gradient, the penalty actor-critic, and the regression gradient, tracked against effective sample size. If our estimator does not track the reference better than the regression gradient, the method's premise fails and we stop.
The second experiment is reasoning under a verifier budget. We use GSM8K with a 7B base model, $G$ equal to exact-match correctness revealed on a fraction $\rho$ of rollouts, and a learned reward model of varying size. We report held-out GSM8K accuracy, the number of $G$ calls, generated tokens, GPU-hours, and peak memory. The decisive check is the interaction between reward-model size and update rule described above.
The third experiment is preference alignment against a limited judge. We train on UltraFeedback and Anthropic-HH prompts. $G$ is a large held-out judge reward model queried under a budget, which gives a controlled over-optimization setting. We evaluate on the Open LLM Leaderboard and RewardBench to allow comparison with AIHF and IRFT, and on held-out judge score to measure reward hacking over training.
Across all settings, the result that would change our minds is simple. If online reward fitting matches us at equal cost everywhere, the note's scientific claim is wrong, even if the computational claim stands.
References
- PARL: A Unified Framework for Policy Alignment in Reinforcement Learning from Human Feedback
- Joint Reward and Policy Learning with Demonstrations and Human Feedback Improves Alignment
- Principal-Driven Reward Design and Agent Policy Alignment via Bilevel-RL
- Bilevel Reinforcement Learning via the Development of Hyper-gradient without Lower-Level Convexity
- A Hessian-Free Actor-Critic Algorithm for Bi-Level Reinforcement Learning with Applications to LLM Fine-Tuning
- On the Sample Complexity Bounds of Bilevel Reinforcement Learning
- ScaleBiO: Scalable Bilevel Optimization for LLM Data Reweighting
- Meta-Reward-Net: Implicitly Differentiable Reward Learning for Preference-based Reinforcement Learning
- On Learning Intrinsic Rewards for Policy Gradient Methods
- Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization
- P-TA: Using Proximal Policy Optimization to Enhance Tabular Data Augmentation via Large Language Models
- Execution-based Code Generation using Deep Reinforcement Learning
- PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training
- Getting More Juice Out of the SFT Data: Reward Learning from Human Demonstration Improves SFT for LLM Alignment
- Penalty Method for Inversion-Free Deep Bilevel Optimization