Standard Reinforcement Learning (RL) is notoriously brittle. If you train a robot to navigate to a specific red button, and then move that button three feet to the left, the agent often fails completely. This is because traditional RL conflates "how the world works" (the physics and layout of the environment) with "what the agent should do" (the reward function). When the goal changes, the agent’s entire understanding of the world—embedded in its value function—becomes obsolete, necessitating expensive retraining from scratch.
A growing body of research suggests a more efficient path: decoupling. By learning the underlying structure of the environment independently of any specific reward, agents can achieve "zero-shot" generalization, adapting to new tasks instantly by simply overlaying a new objective onto a pre-learned map of the world.
The Problem of Entangled Representations
In a typical MDP (Markov Decision Process), the value function $V(s)$ represents the expected sum of future rewards. As Steven Hansen notes in Deep Episodic Value Iteration for Model-based Meta-Reinforcement Learning, this coupling is the primary source of sample inefficiency. Because the reward is "baked into" the value estimate, the agent cannot distinguish between a wall it cannot pass and a goal it simply doesn't want to reach.
This entanglement makes agents "task-specific" rather than "taskable." As León Illanes and Sheila A. McIlraith argue in Symbolic Plans as High-Level Instructions for Reinforcement Learning, humans don't learn to walk specifically to get water; we learn to walk as a general capability and then apply that skill to the goal of getting water.
Successor Features: The Map and the Destination
The most prominent mechanism for this decoupling is the Successor Representation (SR) or its deep-learning evolution, Successor Features (SF).
Instead of learning a single value, an SF-based agent learns to predict the "expected future occupancy" of various features in the environment. If the environment is a maze, the SFs tell the agent which rooms it is likely to visit in the future, given its current policy. Crucially, this prediction is made without knowing what those rooms are "worth."
When a reward function is finally provided, the value of a state is simply the dot product of these Successor Features and a reward weight vector. Christopher Grimm and Andre Barreto demonstrate in Disentangled Cumulants Help Successor Representations Transfer to New Tasks that if these features are "disentangled"—meaning they represent independent aspects of the world like position or color—the agent can solve an exponential number of new tasks almost instantly.
This idea is extended by Chen Ma and Yoshua Bengio in Universal Successor Features for Transfer Reinforcement Learning, which allows agents to leverage the environment's underlying structure to solve entirely unseen goals by decoupling "how the world works" from "what the goal is."
Reward-Free Exploration
If we can solve tasks zero-shot once we have a map, how do we build the map without a reward to guide us? This has led to the rise of Reward-Free Reinforcement Learning.
In this paradigm, the agent undergoes an initial "exploration phase" where its only goal is to gather as much information about the environment's transitions as possible. Emilie Kaufmann and colleagues introduced RF-UCRL, an algorithm that explores to minimize uncertainty about the environment's dynamics. Similarly, Weitong Zhang and Quanquan Gu proved in Reward-Free Model-Based Reinforcement Learning with Linear Function Approximation that an agent can learn a near-optimal policy for any reward function after a single exploration phase.
This "pre-training" phase is analogous to a child playing in a room. By the time the child is asked to "find the blue ball," they already know where the furniture is and how to move their legs; they only need to locate the ball. Archit Sharma and Sergey Levine formalize this in DADS, where agents discover a library of "skills" (predictable changes in the environment) without any external reward, which can then be composed to solve complex tasks.
Scaling to Foundation Agents
The logical conclusion of this research direction is the creation of Foundation Agents (FAs). Just as Large Language Models are pre-trained on text to perform many downstream tasks, FAs are pre-trained on massive interactive datasets to understand the "physics" of decision-making.
Xiaoqian Liu argues in Position: Foundation Agents as the Paradigm Shift for Decision Making that we are moving toward a world where agents are no longer trained for specific tasks but are instead "prompted" with goals. A recent example is PreFeRMAB, developed by YUNFAN ZHAO and Edwin Zhang in Towards a Pretrained Model for Restless Bandits via Multi-arm Generalization. By pre-training on a wide variety of resource allocation problems, PreFeRMAB can be deployed on entirely new tasks with zero-shot performance, bypassing the need for the millions of trials usually required by RL.
Why It Works: The Invariance of Dynamics
The intuition behind this shift is simple: Dynamics are more stable than goals.
The gravity on Earth, the friction of a floor, and the layout of a city do not change just because you decided to go to a different restaurant. By forcing the agent to learn these invariants separately from the transient rewards of a specific mission, we create systems that are not just faster to train, but fundamentally more robust. Whether through Successor Features, Goal-Conditioned architectures, or unsupervised skill discovery, the goal is the same: an agent that understands the world well enough to do whatever it is told.