Standard reinforcement learning (RL) methods typically entangle two distinct pieces of information: how the world works (the environmental transition physics) and what the agent is trying to achieve (the reward function). When an agent learns a policy $\pi(a|s)$ or a value function $Q(s,a)$ end-to-end, it memorizes specific action sequences that maximize a specific reward under a specific set of physical parameters.
This entanglement creates a brittle system. If the task changes (e.g., a robot needs to unstack blocks instead of stacking them) or the environment shifts (e.g., the robot operates on a slippery surface), the learned policy often collapses entirely.
To achieve true zero-shot task adaptation, a growing body of research focuses on decoupling dynamics modeling from task-specific optimization. By isolating the underlying physics into a task-agnostic world model, agents can treat the environment as a reusable simulator, adapting instantly to new goals or physical perturbations at test time without additional training.
Separating Physics from Goals
The core intuition behind zero-shot task adaptation is that the laws of physics remain constant across different tasks within the same environment. If an agent can master the transition dynamics $P(s'|s,a)$ independently of a reward signal, it can evaluate any new reward function instantly via lookahead planning or zero-shot policy generation.
Honghua Dong et al. formalized this concept in Universal Agent for Disentangling Environments and Tasks by introducing a "PATH function" that maps state reachability independently of rewards. This separation allows the agent to leverage its understanding of physics to solve novel tasks without extensive retraining. Similarly, Philip Ball et al. show in Learning General World Models in a Handful of Reward-Free Deployments that training a policy entirely within a pre-trained, reward-free world model allows practitioners to define new rewards at test time, achieving immediate zero-shot transfer.
Arthur Argenson demonstrates in Model-Based Offline Planning that planning directly at test time using a decoupled dynamics model enables an agent to adapt to entirely new constraints and headings with zero retraining. This principle applies even to complex structural shifts in tasks; Yuanlin Duan et al. show in Learning World Models for Unconstrained Goal Navigation that learning the bidirectional connective dynamics between discovered milestones allows agents to perform tasks in reverse (such as unstacking objects after only learning to stack them) without explicit training.
Entangled RL: [State/Action History] ───> [Task-Specific Policy] ───> Brittle Execution
Decoupled RL: [State/Action History] ───> [Task-Agnostic World Model]
│
▼ (Plug-and-Play Reward/Latent)
[Zero-Shot Adaptation / Planning]
This decoupling can also be achieved by decomposing value functions into environmental and reward components. Raymond Chua et al. introduce a streamlined approach to learning these representations directly from pixels in Learning Successor Features the Simple Way, ensuring rapid adaptation when goals shift. To scale this, the authors of Transferable Reinforcement Learning via Generalized Occupancy Models combine successor features with diffusion models to represent the distribution of all possible future outcomes, treating a new reward function as a gradient to bias the sampling process zero-shot.
The benefits of this separation extend beyond single-agent control:
- Multi-Agent Systems: Orr Krupnik et al. decouple world dynamics from agent interaction patterns in Multi Agent Reinforcement Learning with Multi-Step Generative Models., allowing agents to switch between cooperative and competitive behaviors simply by altering the optimization objective during planning.
- Stochastic Environments: Brendan Juba et al. prove in Learning Probably Approximately Complete and Safe Action Models for Stochastic Worlds that learning underlying transition dynamics rather than task-oriented behaviors allows a single model to be safely reused for diverse downstream planning tasks.
- Dynamic Action Spaces: The authors of The Adaptive Q-Network for Recommendation Tasks with Dynamic Item Space shift reinforcement learning from specific item IDs to continuous characteristic spaces, facilitating zero-shot generalization when the available action set changes dynamically.
- Biological Plausibility: Andrea Brovelli demonstrates in A generative spiking neural-network model of goal-directed behaviour and one-step planning that learning a world model independently of specific objectives successfully replicates the flexible, goal-directed behavior observed in humans.
Adapting to Changing Environmental Dynamics
Decoupling does not just protect agents against changes in the goal; it also provides a mechanism to handle variations in the physics of the environment itself, such as changes in mass, friction, or gravity.
When an agent encounters physical perturbations, it can infer the environmental context implicitly from short histories of state-action transitions. Aidan Scannell et al. leverage this mechanism in Residual learning and context encoding for adaptive offline-to-online reinforcement learning to enable successful adaptation to out-of-distribution physical changes. To formalize this transition space, Vaisakh Shaj et al. model system dynamics as a function of a low-dimensional latent variable in Hidden Parameter Recurrent State Space Models For Changing Dynamics Scenarios, allowing a single model to interpolate smoothly between different physical regimes by mapping them to a continuous latent space.
[Short Transition History] ───> [Context Encoder] ───> Latent Physics Parameter (μ)
│
▼
[Context-Conditioned World Model]
Training on a small, discrete set of these physical variations can be surprisingly powerful. Jongseong Chae et al. demonstrate in Robust Imitation Learning against Variations in Environment Dynamics that training an agent on discrete environment perturbations allows the resulting policy to interpolate across the continuous range of unseen dynamics between those points. When physical changes are drastic—such as sudden hardware damage—representing behavior outcomes as relative displacements allows an autonomous system to maintain performance. Konstantinos I. Chatzilygeroudis et al. leverage this in Towards semi-episodic learning for robot damage recovery to enable on-the-fly recovery without re-learning how the physical state affects capabilities.
Furthermore, capturing the underlying dynamics of state-space transitions proves more robust than learning specific trajectory patterns. Andreas Look et al. show in Cheap and Deterministic Inference for Deep State-Space Models of Interacting Dynamical Systems that probabilistic models that analytically propagate uncertainty demonstrate strong zero-shot generalization to novel environments with complex agent interactions.
Building World Models via Task-Agnostic Exploration
How do we train a highly accurate world model across the entire reachable state space without a task reward to guide the agent? The answer lies in task-agnostic active exploration and generative rehearsal.
Pranav Shyam et al. utilize an ensemble of forward dynamics models to drive exploration toward states with high model disagreement in Model-Based Active Exploration, producing a comprehensive world model capable of zero-shot transfer to diverse downstream objectives. Bhavya Sukhija et al. provide theoretical convergence guarantees for this approach in Optimistic Active Exploration of Dynamical Systems, showing that prioritizing global system identification over specific reward-based tasks produces models that are significantly more robust to novel task requirements. As a result, Sai Rajeswar et al. demonstrate in Mastering the Unsupervised Reinforcement Learning Benchmark from Pixels that world models provide a more robust foundation for unsupervised RL than model-free approaches because they capture environment dynamics independently of reward signals.
Once a base understanding of the world is established, agents can "rehearse" or simulate diverse scenarios internally to prepare for unseen environments:
- Generative Rehearsal: Chengxing Jia et al. propose training meta-policies on a diverse set of generated hypothetical environments in Policy Rehearsing: Training Generalizable Policies for Reinforcement Learning, compensating for a total lack of real-world interaction data by preparing the agent for a wide range of potential physical scenarios.
- Data Hallucination: Suzan Ece Ada et al. use a world model to hallucinate training data from a single real-world interaction in Meta-World Conditional Neural Processes, drastically reducing the real-world data required to master new tasks.
- Sequence Modeling: Zhi Wang et al. combine a context-aware world model with a self-guided prompt in Meta-DT: Offline Meta-RL as Conditional Sequence Modeling with World Model Disentanglement to achieve zero-shot inference without expert demonstrations at test time.
Modern approaches also leverage foundation models or non-curated datasets to bootstrap this understanding. Wenlong Huang et al. use zero-shot trajectories generated by large language models as initial priors to accelerate dynamics learning in VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models. Yann LeCun demonstrates in OSVI-WM: One-Shot Visual Imitation for Unseen Tasks using World-Model-Guided Trajectory Generation that specialized world models can achieve high-performance one-shot visual imitation on unseen tasks without massive internet-scale pretraining, proving that internal reasoning about environment dynamics is a viable alternative to data-intensive pre-training.
This foundational understanding can be built from highly imperfect data. The authors of Efficient Reinforcement Learning by Guiding World Models with Non-Curated Data demonstrate that world models pre-trained on broad, non-curated data enable significantly faster adaptation to sequences of increasingly complex tasks compared to traditional incremental learning methods. Similarly, Arunkumar Byravan et al. pre-train a shared world model across diverse source tasks in Imagined Value Gradients: Model-Based Policy Optimization with Tranferable Latent Dynamics Models, freezing the latent dynamics to enable rapid adaptation to new target objectives by reusing the established understanding of physical interactions.
Test-Time Planning and In-Context Adaptation
With a decoupled world model in hand, how does the agent execute actions for a new task? Instead of relying on a fixed, pre-trained policy, the agent plans dynamically at test time, using the world model to evaluate potential futures.
Haoru Xue et al. implement real-time optimization using full-order physics models in Full-Order Sampling-Based MPC for Torque-Level Locomotion Control via Diffusion-Style Annealing, providing immediate zero-shot adaptation to payloads or surface friction where standard RL policies fail. Yuxiang Yang et al. demonstrate a similar capability in Data Efficient Reinforcement Learning for Legged Robots., showing that a task-agnostic dynamics model allows quadrupeds to adapt to new directions or speeds zero-shot by simply updating the reward function during the planning phase.
Test-Time Execution Loop:
┌────────────────────────────────────────────────────────┐
│ │
▼ │
[Current State] ───> [Simulate Futures via World Model] ─┤
│ │
▼ │
[Score via New Reward] │
│ │
▼ │
[Execute Optimal Action] ───────────┘
For visual environments where planning is traditionally difficult, Sampada Deglurkar et al. combine generative models with online Monte Carlo Tree Search in Compositional Learning-based Planning for Vision POMDPs, ensuring the agent's physical understanding remains valid even when goals or environmental constraints change. Clement Gehring shows that combining low-rank matrix approximations with trajectory optimization enables fast, adaptable replanning in continuous spaces in Adaptable replanning with compressed linear action models for learning from demonstrations.
When explicit tree search is too expensive, model-based planning can be offloaded to modular architectures or sequence models. The authors of Learning and Planning Multi-Agent Tasks via an MoE-based World Model use a modular Mixture-of-Experts world model to simulate future outcomes, allowing agents to derive successful action sequences for novel task configurations without a generalized policy. Alternatively, Jinmei Liu et al. introduce a multi-head transformer architecture in Scalable In-Context Q-Learning to predict values and policies simultaneously, leveraging the world model to stitch together suboptimal experiences and generalize zero-shot to out-of-distribution target velocities.
Ultimately, this paradigm shift from "learning a policy" to "learning a world model" changes the role of the reward function. As Weichao Zhou et al. argue in Rethinking Inverse Reinforcement Learning: from Data Alignment to Task Alignment, treating expert data as a means to identify a set of plausible task-aligned rewards rather than a single rigid policy ensures that the resulting behaviors remain robust and transferable across environments with wildly varying dynamics.
Key Foundational Work
Neural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuning (2017) — contributed 1 concept elements to this direction.