Disentangling Reward Functions from Environment Dynamics

This direction is merged into another direction: cross-environment reward transfer. Its last version stays readable and citable here.

Version 1 of 142 papers65 conceptsfrozen 2026-10-04

When we teach a robot to perform a task, we often face a choice: do we show it how to move, or do we tell it what to achieve? Standard imitation learning, such as behavioral cloning, focuses on the "how." By mimicking the exact joint velocities or actions of an expert, an agent can quickly learn a specific trajectory. However, this approach is notoriously brittle. If the robot’s mass changes, a joint stiffens, or the floor friction varies, the learned policy collapses because the specific actions required to reach the goal have changed.

To build agents that generalize, researchers have turned to Inverse Reinforcement Learning (IRL). The core insight, as explored by Chelsea Finn and Sergey Levine in Generalizing Skills with Semi-Supervised Reinforcement Learning, is that while optimal actions depend heavily on physics, the underlying reward function—the "intent"—is often invariant to environment dynamics.

The Entanglement Problem

The challenge with traditional IRL is that the recovered reward functions often become "entangled" with the specific physics of the training environment. If an agent learns a reward function while navigating a high-friction surface, that reward might implicitly include "shaping" terms that compensate for that friction. When transferred to a low-friction environment, this reward function no longer represents the true goal, leading to suboptimal behavior.

Justin Fu and colleagues identified this in Learning Robust Rewards with Adversarial Inverse Reinforcement Learning. They noted that for a reward to be truly portable, it must be decoupled from the transition dynamics of the environment. Without this disentanglement, the reward is just another form of overfitted policy.

Achieving Disentanglement via State-Only Rewards

A primary mechanism for achieving this separation is restricting the reward function's domain. In Learning Robust Rewards with Adversarial Inverse Reinforcement Learning, the authors demonstrate that defining rewards solely as a function of state ($R(s)$), rather than state-action pairs ($R(s, a)$), significantly improves robustness.

Why does this work? Actions are the interface between the agent and the physics of the world. By removing actions from the reward definition, we force the model to focus on the outcomes (the states) rather than the mechanics used to reach them. This concept is echoed in A Primer on Maximum Causal Entropy Inverse Reinforcement Learning by Adam Gleave and Sam Toyer, who argue that state-only rewards are more likely to generalize across different physical configurations or transition rules.

Adversarial Training and Potential Functions

The state-of-the-art approach to this problem is Adversarial Inverse Reinforcement Learning (AIRL). AIRL uses a discriminator to distinguish between expert demonstrations and agent trajectories, similar to a Generative Adversarial Network (GAN).

To ensure the reward is disentangled, AIRL structures the discriminator's reward signal into two parts:

  1. A core reward function $g(s)$ that represents the true objective.
  2. A "shaping" or potential function $h(s)$ that accounts for environment-specific nuances.

By isolating $g(s)$, the system recovers a reward that is "dynamics-invariant." This allows the learned intent to be transferred to entirely different agents. For instance, Tianwei Ni and Harshit S. Sikchi showed in $f$-IRL: Inverse Reinforcement Learning via State Marginal Matching that rewards focused on state distributions allow a reward optimized for one robot morphology to successfully guide an agent with a completely different body structure.

Robustness to Distribution Shift and Non-Stationarity

The benefits of disentangled rewards extend beyond physical changes. In financial contexts, Jacobo Roa-Vicens and Yarin Gal applied AIRL to market data in Adversarial recovery of agent rewards from latent spaces of the limit order book. They found that recovering the underlying reward function allowed agents to adapt to shifting market regimes (like changing volatility) where direct imitation would fail.

Furthermore, Ivan Ovinnikov introduced the causal invariance principle in Learning Causally Invariant Reward Functions from Diverse Demonstrations. By treating demonstrations from different environments as "interventions," the learning process can ignore spurious correlations—features that appear relevant in one setting but don't cause the expert's success—further purifying the reward signal.

Why It Works: The Transferability of Intent

The fundamental reason this research direction succeeds is that intent is more stable than behavior.

As Alberto Maria Metelli proves in Provably Efficient Learning of Transferable Rewards, IRL recovers the "why" of a task. When an agent understands the "why," it can use a standard Reinforcement Learning algorithm to figure out the "how" in any new environment. Whether it is a robot adapting to a broken limb or a reasoning model guiding a search process (Your Reward Function for RL is Your Best PRM for Search), disentangled rewards provide a universal language for task objectives that survives the transition from training to the real, unpredictable world.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_disentangling_reward_functions_from_envi_7940cac8,
  title = {Disentangling Reward Functions from Environment Dynamics},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/disentangling-reward-functions-from-environment-dynamics/rdr_7940cac840744a59a0f047abec0e1908},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_disentangling_reward_functions_from_envi_7940cac8_v1,
  title = {Disentangling Reward Functions from Environment Dynamics},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/disentangling-reward-functions-from-environment-dynamics/rdr_7940cac840744a59a0f047abec0e1908/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}