When an agent learns to navigate a maze, drive a car, or control a robotic arm via standard reinforcement learning (RL), it maps states and actions directly to expected long-term returns. In doing so, the agent implicitly glues two fundamentally distinct pieces of information together: what it is trying to achieve (the reward function) and how the world responds to its movements (the transition dynamics).
This entanglement makes deep RL notoriously brittle. If you change the environment's physics—such as altering surface friction, introducing a hardware fault, or crippling a robot's joint—the entire policy collapses. Even though the high-level goal remains identical, the agent cannot adapt because its understanding of the goal is structurally fused with its old physical capabilities. Conversely, if the physics remain constant but the goal changes, the agent must typically relearn its policy from scratch.
To build agents that truly generalize, a growing body of research focuses on dynamics-reward decoupling. By separating intent from execution, we can create modular components that transfer seamlessly across different environments and tasks.
The Core Intuition of Disentanglement
Consider a standard Markov Decision Process (MDP) defined by transition probabilities $P(s'|s,a)$ and a reward function $R(s,a)$. The transitions represent the laws of physics, while the rewards represent the task objective.
Traditional end-to-end RL algorithms obscure this boundary by learning a monolithic policy $\pi(a|s)$ or value function $Q(s,a)$ that depends on both $P$ and $R$ simultaneously. If either changes, the entire network must be retrained.
Decoupling architectures break this dependency by forcing the agent to learn separate, modular representations for physics and goals. This factorization unlocks two powerful transfer paradigms:
- Cross-Environment Transfer: A reward function learned from an expert in a source environment can be exported to a target environment with entirely different physics, allowing a new agent to learn the same task without needing new demonstrations.
- Cross-Task Transfer: An agent can maintain a stable, task-agnostic model of the environment's physics, allowing it to instantly solve new objectives without re-learning how the world moves.
Extracting Environment-Agnostic Intent
When learning from expert demonstrations (Inverse Reinforcement Learning, or IRL), the goal is to recover the underlying reward function that motivated the expert. However, if the expert drives a high-performance vehicle, a naive IRL algorithm might output a reward function that penalizes slow acceleration—confounding the expert's intent with the vehicle's mechanics.
To isolate pure intent, Justin Fu et al. proposed Adversarial Inverse Reinforcement Learning (AIRL) in Learning Robust Rewards with Adverserial Inverse Reinforcement Learning. They demonstrated that by structuring the reward function to depend exclusively on states rather than specific actions, the recovered objective becomes invariant to changes in environment dynamics. If a robot's joint is disabled, an agent equipped with this disentangled reward function can automatically discover alternative physical strategies to achieve the exact same high-level goal.
Standard IRL: [State, Action] ───> [Dynamics-Dependent Reward]
Disentangled IRL: [State Transitions] ───> [Pure Intent Reward] (Transferable)
Taking this a step further, Tianwei Ni et al. introduced $f$-IRL in f-IRL: Inverse Reinforcement Learning via State Marginal Matching. Instead of matching specific state-action trajectories, their framework extracts rewards by matching the expert's state marginal distribution—essentially focusing on where the agent should be rather than how it gets there. This allows a reward function optimized on one robotic morphology to successfully guide an agent with a completely different physical body.
As Michael Herman et al. showed in Inverse Reinforcement Learning with Simultaneous Estimation of Rewards and Dynamics, explicitly modeling the system's physics alongside the reward function prevents environmental constraints from leaking into the objective parameters, ensuring robust transfer to novel layouts.
Reusing Dynamics via Successor Features
The converse problem is equally vital: if the environment's physics are invariant but the goals change frequently, how can an agent adapt instantly?
A powerful mechanism for this is Successor Features (SF), which mathematically factorizes the value function into a dynamics-dependent basis and a task-dependent weight vector. The basis predicts the discounted future occupancy of environmental features, while the weights represent the immediate rewards of those features.
Chen Ma demonstrated in Universal Successor Representations for Transfer Reinforcement Learning that this decoupling allows agents to retain a stable representation of the world's physics across changing objectives. When a goal shifts, the agent does not need to re-simulate or re-explore how states connect; it merely updates its linear reward mapping.
Value Function Q(s,a) = ψ(s,a) [Successor Features: Physics/Occupancy]
× w [Weight Vector: Task-Specific Reward]
This separation enables rapid, often zero-shot adaptation. Boyuan Chen et al. showed in Self-Supervised Reinforcement Learning that Transfers using Random Features that isolating environment-specific dynamics into a value function basis allows an agent to master out-of-distribution tasks in a single step, without updating the underlying neural network parameters.
Furthermore, Angelos Filos et al. leveraged this concept in PsiPhi-Learning: Reinforcement Learning with Demonstrations using Successor Features and Inverse Temporal Difference Learning to allow agents to learn from diverse, unlabeled demonstrators. Because the framework separates world mechanics from individual preferences, an agent can safely acquire mechanical knowledge (e.g., how a door opens) from a demonstrator, even if that demonstrator has a completely conflicting ultimate goal.
Actively Bridging the Dynamics Gap
In many practical scenarios, such as transferring a policy from a simulator to a real-world robot, the reward function itself must be adjusted to actively compensate for differences in physics.
Instead of trying to make the reward completely blind to dynamics, Benjamin Eysenbach et al. developed DARC in Off-Dynamics Reinforcement Learning: Training for Transfer with Domain Classifiers. DARC uses a binary classifier to identify transitions that are physically impossible or highly divergent between the source and target domains. It then modifies the source reward function to penalize those transitions, steering the agent toward behaviors that are inherently stable and transferable across domains.
Similarly, Zixuan Liu et al. introduced DYNAIL in Dynamics Adapted Imitation Learning, which employs a dynamics-aware regularization term to penalize physically inconsistent transitions. By decoupling high-level intent from low-level execution parameters, the agent avoids the trap of sub-optimal local optima caused by trying to mimic exact trajectory paths that its target hardware cannot physically support.
By breaking the monolithic structure of traditional reinforcement learning into isolated, specialized modules for rewards and dynamics, these frameworks ensure that what an agent learns about the world and what it learns about its goals remain robust, reusable, and resilient to change.