Inverting Policy Dynamics for Reward Inference

19 concepts7 papers1 proposal

How do we figure out what an agent is trying to achieve? Standard Inverse Reinforcement Learning (IRL) operates under a strict assumption: we observe a static, perfectly optimal expert, and we find a reward function that makes their actions look optimal. However, this assumption breaks down in two common real-world scenarios. First, the agent we are observing might not be an expert yet; they might be a student actively learning and modifying their behavior over time. Second, we ourselves might have a pre-trained, sub-optimal policy (e.g., from behavioral cloning) that we want to fine-tune using reinforcement learning, but we lack a well-aligned reward function to guide the transition without destroying what the policy already knows.

A powerful paradigm addresses both challenges by inverting policy dynamics. Instead of treating the relationship between a policy and a reward function as a one-way street—where a reward generates a policy via forward reinforcement learning—this line of research mathematically inverts policy updates, gradients, and structures to infer the underlying reward function directly.

Inferring Goals from the Trajectory of Learning

When an agent learns, its policy changes. If we assume the agent is improving its behavior via some optimization process, the difference between its consecutive policies contains a rich signal about its objective. Imagine watching someone practice archery: if their first arrow hits too low and they adjust their aim upward on the next shot, their adjustment reveals their goal just as clearly as a perfect bullseye would.

This insight forms the foundation of work by Alexis D. Jacq et al. in Learning from a Learner. Rather than requiring demonstrations from a fully proficient expert, their framework extracts reward functions by analyzing the performance gains between successive versions of a learning agent's policy. By comparing two consecutive policy snapshots from an entropy-regularized learner, a mathematical framework called soft policy improvement inversion recovers the reward function. The core intuition is that the change in the agent's action distribution directly reflects an underlying optimization toward a specific objective; we can deduce the goal by observing how the agent corrects its mistakes over time.

Building on this idea, Giorgia Ramponi et al. introduced the LOGEL algorithm in Inverse Reinforcement Learning from a Gradient-based Learner. Instead of looking at the magnitude of performance gains, they model the agent's policy updates explicitly as steps along the gradient of its expected return. By analyzing the direction of parameter changes over time, the observer can infer the agent's goals even if the learning trajectory is noisy or non-monotonic. This makes reward recovery robust against temporary mistakes made by the learner.

This principle also scales to multi-agent environments where no single agent has reached a stable equilibrium. Mine Melodi Caliskan et al. extended this paradigm in Multi-Agent Learning from Learners, demonstrating that the transition between consecutive policies during a multi-agent learning process provides a sufficient signal to recover an agent's underlying reward function. Interestingly, the authors showed that the accuracy of the recovered reward function scales logarithmically relative to the performance improvement of the observed learner, meaning that as the learner refines its policy, the quality of the inferred reward increases substantially.

Differentiable Planning Layers

Another way to invert the policy-reward relationship is to embed the entire planning process into a differentiable architecture. Traditional IRL often assumes a specific, rational planning model, which fails when humans exhibit systematic biases.

To address this, Rohin Shah et al. investigated backpropagating through a differentiable planning layer in On the Feasibility of Learning, Rather than Assuming, Human Biases for Reward Inference. By making the planner end-to-end differentiable, the underlying reward parameters can be updated based on how well the planner's output aligns with observed demonstrations. This allows the system to learn human planning biases directly from data rather than hard-coding them, minimizing errors introduced by incorrect behavioral assumptions.

Preventing Policy Collapse via Coherent Inversion

The mathematical inversion of policies and rewards is not just useful for observing external agents; it is also a powerful tool for stabilizing our own training pipelines. A notorious problem arises when fine-tuning a pre-trained imitation policy (such as one learned via behavioral cloning) using reinforcement learning. As noted by Abhishek Gupta et al. in Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learning, the agent often suffers from an immediate degradation or "unlearning" of the demonstrated behaviors. This collapse occurs because the initial, unaligned reward signals and value estimates conflict with the pre-trained policy, forcing the agent to discard its expert knowledge during early exploration before it can discover a better strategy.

To solve this, Joe Watson et al. proposed Coherent Soft Imitation Learning (CSIL) in Coherent Soft Imitation Learning and Coherent Soft Imitation Learning. Instead of imposing an arbitrary initial reward function, CSIL analytically inverts the entropy-regularized Bellman equation to derive a shaped reward function for which the pre-trained behavioral cloning policy is already optimal by definition. Specifically, this reward is defined as the log-difference between the learned policy and a stationary prior distribution. Because the policy begins at a local optimum with respect to this derived reward, the transition from imitation to reinforcement learning is perfectly seamless, entirely preventing the initial performance drop.

Standard RL Fine-Tuning:
[Pre-trained Policy] + [Unaligned Initial Reward] ──> Exploration Disconnect ──> Policy Collapse (Unlearning)

Coherent Soft Inversion:
[Pre-trained Policy] ──> Invert Bellman Equation ──> [Coherent Shaped Reward] ──> Stable Local Optimum ──> Smooth Improvement

This framework scales effectively to large-scale models. In Coherent Off-Policy Improvement of Large Behaviour Models with Learned Rewards, the authors leveraged a similar mathematical alignment—deriving a reward function from the log-likelihood of a pre-trained policy relative to a uniform distribution—to fine-tune Large Behaviour Models (LBMs). By ensuring that the initial policy and critic are perfectly coherent, the framework eliminates the initial performance collapse and enables highly sample-efficient fine-tuning on complex, sparse-reward robotic tasks.

Whether used to track a student's progress or to anchor a pre-trained model during fine-tuning, policy-reward inversion changes the fundamental assumption of inverse reinforcement learning. By treating the policy-reward relationship as an invertible mathematical bridge, these methods allow us to read the implicit objectives written directly into the structure and updates of neural policies.

Key Foundational Work

Learning from a Learner (2019) — contributed 3 concept elements to this direction.
Inverse Reinforcement Learning from a Gradient-based Learner (2020) — contributed 2 concept elements to this direction.
On the Feasibility of Learning, Rather than Assuming, Human Biases for Reward Inference (2019) — contributed 1 concept elements to this direction.
History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 8 papers
  2. Version 1 · 2026-10-04 · 10 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_gradient_based_reward_inference_aa570974,
  title = {gradient-based reward inference},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/gradient-based-reward-inference/rdr_aa570974fa1c4b878ea9e5dbaa91e281},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 2, frozen text):

@misc{lacuna_gradient_based_reward_inference_aa570974_v2,
  title = {gradient-based reward inference},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/gradient-based-reward-inference/rdr_aa570974fa1c4b878ea9e5dbaa91e281/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake