Bilevel Optimization for Policy Alignment and Reward Design

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 2117 papers139 conceptsfrozen 2026-10-04

In Reinforcement Learning (RL), we often face a "specification gap." We want an agent to behave safely and efficiently, but we can only provide it with a mathematical reward function. When an agent optimizes this reward, it frequently discovers "hacks"—behaviors that maximize the numerical score while violating our actual intent.

Recent research has converged on a powerful solution: treating reward design not as a static engineering task, but as a bilevel optimization problem. By nesting the agent's learning process inside a higher-level optimization loop, we can create reward functions that are "aware" of how an agent will react to them.

The Nested Loop of Learning

The core intuition behind this research direction is a shift from a single-level objective to a two-tier hierarchy:

  1. The Inner Loop (Policy Optimization): The agent learns a policy $\pi$ to maximize a given reward function $R_\theta$. This is standard reinforcement learning.
  2. The Outer Loop (Reward Optimization): A "meta-optimizer" adjusts the parameters $\theta$ of the reward function. It evaluates the resulting policy from the inner loop against a high-level goal—such as human preferences, expert demonstrations, or sparse task success.

As Aleksandra Faust and colleagues demonstrate in Evolving Rewards to Automate Reinforcement Learning, this structure allows us to automate the "black art" of reward shaping. Instead of manually tuning weights, an evolutionary outer loop can search for reward parameters that lead to the best final performance on complex tasks like humanoid locomotion.

Solving the Distribution Shift in RLHF

A major challenge in Reinforcement Learning from Human Feedback (RLHF) is that the reward model is often trained on a static dataset of human preferences. However, as the agent learns, it explores new states that the original reward model hasn't seen. This leads to "reward overoptimization," where the agent exploits the reward model's uncertainty.

Amrit Bedi, Souradip Chakraborty, and their team address this in PARL: A Unified Framework for Policy Alignment in Reinforcement Learning from Human Feedback. By framing alignment as a bilevel problem, they explicitly model the "entanglement" between the reward function and the evolving policy. The outer loop adjusts the reward specifically to keep the agent aligned even as it moves into new regions of the state space, achieving significantly better sample efficiency and robustness against reward hacking.

Differentiating Through the Learning Process

How does the outer loop know how to change the reward? The most sophisticated methods use meta-gradients. By calculating the gradient of the final performance with respect to the reward parameters, the system can "see" how a small change in the reward will ripple through the agent's entire learning trajectory.

  • In Multi-Agent Systems: Jiachen Yang introduces Learning to Incentivize Other Learning Agents (LIO), where agents learn to give supplementary rewards to peers. The "giver" differentiates through the "recipient's" policy update to ensure the incentives actually lead to mutually beneficial behavior.
  • In Reward Shaping: Yujing Hu and others developed BiPaRS, which uses meta-gradients to weight human-provided "hints." If a hint is misleading, the bilevel optimizer detects that the resulting policy is performing poorly and automatically reduces that hint's influence.

Recovering Intent via Inverse RL

Bilevel optimization is also the engine behind modern Inverse Reinforcement Learning (IRL). In this context, the outer loop's goal is to find a reward function that makes the observed expert demonstrations look optimal.

Siliang Zeng and Alfredo Garcia have advanced this through Offline ML-IRL. Their framework maximizes the statistical likelihood of expert trajectories in the outer loop while solving for the optimal policy in the inner loop. This ensures the recovered reward is mathematically consistent with the expert's behavior, even in offline settings where the agent cannot interact with the environment to test its theories.

Beyond Rewards: Synthetic Environments and Critics

The bilevel approach isn't limited to reward functions; it can optimize the entire training context. Fabio Ferreira explores Learning Synthetic Environments, where the outer loop evolves a "proxy environment" or "training camp." The agent learns faster in this synthetic world because the bilevel optimizer has distilled the target task's essential dynamics into a more dense and informative representation.

Similarly, Sarah Bechtle demonstrates in Model Based Meta Learning of Critics that we can meta-learn the "critic" (the loss function) itself. By differentiating through a dynamics model, the outer loop creates a critic that provides more accurate gradients to the inner-loop policy, leading to faster and more stable convergence.

Why It Works: Modeling the Learner's Response

The fundamental reason bilevel optimization succeeds where manual design fails is anticipation. Traditional RL treats the reward as a fixed landscape. Bilevel RL treats the reward as a steerable parameter that accounts for the agent's learning dynamics. By acknowledging that the agent is a "moving target" that will adapt to any signal we provide, these frameworks allow us to design incentives that guide agents toward true alignment rather than just high scores.

Papers in this version

Showing 60 of 117 papers.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 94 papers
  2. Version 1 · 2026-10-04 · 117 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_bi_level_reward_optimization_4128f887,
  title = {bi-level reward optimization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/bi-level-reward-optimization/rdr_4128f88749844374b8c98ab0bee671f0},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_bilevel_optimization_for_policy_alignmen_4128f887_v1,
  title = {Bilevel Optimization for Policy Alignment and Reward Design},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/bilevel-optimization-for-policy-alignment-and-reward-design/rdr_4128f88749844374b8c98ab0bee671f0/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}