Decoupling Dynamics from Rewards for Zero Shot Adaptation

92 concepts82 papers

Standard reinforcement learning (RL) methods typically entangle two distinct pieces of information: how the world works (the environmental transition physics) and what the agent is trying to achieve (the reward function). When an agent learns a policy $\pi(a|s)$ or a value function $Q(s,a)$ end-to-end, it memorizes specific action sequences that maximize a specific reward under a specific set of physical parameters.

This entanglement creates a brittle system. If the task changes (e.g., a robot needs to unstack blocks instead of stacking them) or the environment shifts (e.g., the robot operates on a slippery surface), the learned policy often collapses entirely.

To achieve true zero-shot task adaptation, a growing body of research focuses on decoupling dynamics modeling from task-specific optimization. By isolating the underlying physics into a task-agnostic world model, agents can treat the environment as a reusable simulator, adapting instantly to new goals or physical perturbations at test time without additional training.

Separating Physics from Goals

The core intuition behind zero-shot task adaptation is that the laws of physics remain constant across different tasks within the same environment. If an agent can master the transition dynamics $P(s'|s,a)$ independently of a reward signal, it can evaluate any new reward function instantly via lookahead planning or zero-shot policy generation.

Honghua Dong et al. formalized this concept in Universal Agent for Disentangling Environments and Tasks by introducing a "PATH function" that maps state reachability independently of rewards. This separation allows the agent to leverage its understanding of physics to solve novel tasks without extensive retraining. Similarly, Philip Ball et al. show in Learning General World Models in a Handful of Reward-Free Deployments that training a policy entirely within a pre-trained, reward-free world model allows practitioners to define new rewards at test time, achieving immediate zero-shot transfer.

Arthur Argenson demonstrates in Model-Based Offline Planning that planning directly at test time using a decoupled dynamics model enables an agent to adapt to entirely new constraints and headings with zero retraining. This principle applies even to complex structural shifts in tasks; Yuanlin Duan et al. show in Learning World Models for Unconstrained Goal Navigation that learning the bidirectional connective dynamics between discovered milestones allows agents to perform tasks in reverse (such as unstacking objects after only learning to stack them) without explicit training.

Entangled RL:  [State/Action History] ───> [Task-Specific Policy] ───> Brittle Execution

Decoupled RL: [State/Action History] ───> [Task-Agnostic World Model] 
                                                    │
                                                    ▼ (Plug-and-Play Reward/Latent)
                                          [Zero-Shot Adaptation / Planning]

This decoupling can also be achieved by decomposing value functions into environmental and reward components. Raymond Chua et al. introduce a streamlined approach to learning these representations directly from pixels in Learning Successor Features the Simple Way, ensuring rapid adaptation when goals shift. To scale this, the authors of Transferable Reinforcement Learning via Generalized Occupancy Models combine successor features with diffusion models to represent the distribution of all possible future outcomes, treating a new reward function as a gradient to bias the sampling process zero-shot.

The benefits of this separation extend beyond single-agent control:

Adapting to Changing Environmental Dynamics

Decoupling does not just protect agents against changes in the goal; it also provides a mechanism to handle variations in the physics of the environment itself, such as changes in mass, friction, or gravity.

When an agent encounters physical perturbations, it can infer the environmental context implicitly from short histories of state-action transitions. Aidan Scannell et al. leverage this mechanism in Residual learning and context encoding for adaptive offline-to-online reinforcement learning to enable successful adaptation to out-of-distribution physical changes. To formalize this transition space, Vaisakh Shaj et al. model system dynamics as a function of a low-dimensional latent variable in Hidden Parameter Recurrent State Space Models For Changing Dynamics Scenarios, allowing a single model to interpolate smoothly between different physical regimes by mapping them to a continuous latent space.

[Short Transition History] ───> [Context Encoder] ───> Latent Physics Parameter (μ)
                                                               │
                                                               ▼
                                                [Context-Conditioned World Model]

Training on a small, discrete set of these physical variations can be surprisingly powerful. Jongseong Chae et al. demonstrate in Robust Imitation Learning against Variations in Environment Dynamics that training an agent on discrete environment perturbations allows the resulting policy to interpolate across the continuous range of unseen dynamics between those points. When physical changes are drastic—such as sudden hardware damage—representing behavior outcomes as relative displacements allows an autonomous system to maintain performance. Konstantinos I. Chatzilygeroudis et al. leverage this in Towards semi-episodic learning for robot damage recovery to enable on-the-fly recovery without re-learning how the physical state affects capabilities.

Furthermore, capturing the underlying dynamics of state-space transitions proves more robust than learning specific trajectory patterns. Andreas Look et al. show in Cheap and Deterministic Inference for Deep State-Space Models of Interacting Dynamical Systems that probabilistic models that analytically propagate uncertainty demonstrate strong zero-shot generalization to novel environments with complex agent interactions.

Building World Models via Task-Agnostic Exploration

How do we train a highly accurate world model across the entire reachable state space without a task reward to guide the agent? The answer lies in task-agnostic active exploration and generative rehearsal.

Pranav Shyam et al. utilize an ensemble of forward dynamics models to drive exploration toward states with high model disagreement in Model-Based Active Exploration, producing a comprehensive world model capable of zero-shot transfer to diverse downstream objectives. Bhavya Sukhija et al. provide theoretical convergence guarantees for this approach in Optimistic Active Exploration of Dynamical Systems, showing that prioritizing global system identification over specific reward-based tasks produces models that are significantly more robust to novel task requirements. As a result, Sai Rajeswar et al. demonstrate in Mastering the Unsupervised Reinforcement Learning Benchmark from Pixels that world models provide a more robust foundation for unsupervised RL than model-free approaches because they capture environment dynamics independently of reward signals.

Once a base understanding of the world is established, agents can "rehearse" or simulate diverse scenarios internally to prepare for unseen environments:

Modern approaches also leverage foundation models or non-curated datasets to bootstrap this understanding. Wenlong Huang et al. use zero-shot trajectories generated by large language models as initial priors to accelerate dynamics learning in VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models. Yann LeCun demonstrates in OSVI-WM: One-Shot Visual Imitation for Unseen Tasks using World-Model-Guided Trajectory Generation that specialized world models can achieve high-performance one-shot visual imitation on unseen tasks without massive internet-scale pretraining, proving that internal reasoning about environment dynamics is a viable alternative to data-intensive pre-training.

This foundational understanding can be built from highly imperfect data. The authors of Efficient Reinforcement Learning by Guiding World Models with Non-Curated Data demonstrate that world models pre-trained on broad, non-curated data enable significantly faster adaptation to sequences of increasingly complex tasks compared to traditional incremental learning methods. Similarly, Arunkumar Byravan et al. pre-train a shared world model across diverse source tasks in Imagined Value Gradients: Model-Based Policy Optimization with Tranferable Latent Dynamics Models, freezing the latent dynamics to enable rapid adaptation to new target objectives by reusing the established understanding of physical interactions.

Test-Time Planning and In-Context Adaptation

With a decoupled world model in hand, how does the agent execute actions for a new task? Instead of relying on a fixed, pre-trained policy, the agent plans dynamically at test time, using the world model to evaluate potential futures.

Haoru Xue et al. implement real-time optimization using full-order physics models in Full-Order Sampling-Based MPC for Torque-Level Locomotion Control via Diffusion-Style Annealing, providing immediate zero-shot adaptation to payloads or surface friction where standard RL policies fail. Yuxiang Yang et al. demonstrate a similar capability in Data Efficient Reinforcement Learning for Legged Robots., showing that a task-agnostic dynamics model allows quadrupeds to adapt to new directions or speeds zero-shot by simply updating the reward function during the planning phase.

Test-Time Execution Loop:
┌────────────────────────────────────────────────────────┐
│                                                        │
▼                                                        │
[Current State] ───> [Simulate Futures via World Model] ─┤
                                 │                       │
                                 ▼                       │
                     [Score via New Reward]              │
                                 │                       │
                                 ▼                       │
                     [Execute Optimal Action] ───────────┘

For visual environments where planning is traditionally difficult, Sampada Deglurkar et al. combine generative models with online Monte Carlo Tree Search in Compositional Learning-based Planning for Vision POMDPs, ensuring the agent's physical understanding remains valid even when goals or environmental constraints change. Clement Gehring shows that combining low-rank matrix approximations with trajectory optimization enables fast, adaptable replanning in continuous spaces in Adaptable replanning with compressed linear action models for learning from demonstrations.

When explicit tree search is too expensive, model-based planning can be offloaded to modular architectures or sequence models. The authors of Learning and Planning Multi-Agent Tasks via an MoE-based World Model use a modular Mixture-of-Experts world model to simulate future outcomes, allowing agents to derive successful action sequences for novel task configurations without a generalized policy. Alternatively, Jinmei Liu et al. introduce a multi-head transformer architecture in Scalable In-Context Q-Learning to predict values and policies simultaneously, leveraging the world model to stitch together suboptimal experiences and generalize zero-shot to out-of-distribution target velocities.

Ultimately, this paradigm shift from "learning a policy" to "learning a world model" changes the role of the reward function. As Weichao Zhou et al. argue in Rethinking Inverse Reinforcement Learning: from Data Alignment to Task Alignment, treating expert data as a means to identify a set of plausible task-aligned rewards rather than a single rigid policy ensures that the resulting behaviors remain robust and transferable across environments with wildly varying dynamics.

Key Foundational Work

Neural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuning (2017) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_zero_shot_task_adaptation_18dec64f,
  title = {zero-shot task adaptation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/zero-shot-task-adaptation/rdr_18dec64f9b444246b745dab4af20215a},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_zero_shot_task_adaptation_18dec64f_v1,
  title = {zero-shot task adaptation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/zero-shot-task-adaptation/rdr_18dec64f9b444246b745dab4af20215a/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake