Fast Adaptation via Policy-Dynamics Value Functions

Roberta Raileanu, Maxwell Goldstein, Arthur Szlam, Rob Fergus

Abstract

Standard reinforcement learning (RL) agents are often brittle, failing when environment dynamics—such as friction or gravity—deviate from training conditions. This paper proposes Policy-Dynamics Value Functions (PD-VF), a framework designed for rapid adaptation to novel dynamics. By conditioning a value function on latent embeddings of both agent policies and environment physics, PD-VF allows for the analytical derivation of optimal policies. Experiments across several continuous control tasks demonstrate that PD-VF can adapt to unseen environments in as few as one to four interaction steps, significantly outperforming existing meta-RL benchmarks.

Introduction

Reinforcement learning has achieved remarkable success in static environments, but real-world deployment requires agents to handle unpredictable changes in physics. For instance, a robot walking on grass must quickly adjust its gait when moving onto ice. Traditional adaptation methods often require extensive re-training or multiple gradient updates, which are too slow for time-critical applications. PD-VF addresses this by framing adaptation as a fast "policy calculation" rather than a slow optimization search.

Methodology

The PD-VF framework decomposes the adaptation problem into four distinct phases:

1. Reinforcement Learning Phase

A library of policies is generated by training an ensemble of agents using Proximal Policy Optimization (PPO) across various environments with diverse dynamics. Checkpoints are saved throughout training to capture a wide range of behaviors, from novice to expert.

2. Self-Supervised Embedding Phase

To represent policies and dynamics in a latent space, the authors employ Transformer-based encoders:

  • Dynamics Encoder ($E_d$): Learns a representation $z_d$ by predicting the next state $s_{t+1}$ given the current state and action.
  • Policy Encoder ($E_\pi$): Learns a representation $z_\pi$ by predicting the actions $a_t$ taken by a specific policy.
Architecture of the policy and dynamics encoders.

3. Supervised Training Phase

The core of the method is the training of a value function $W(s_0, z_\pi, z_d)$ that predicts the expected return $\hat{G}$. To enable fast optimization, $W$ is constrained to a quadratic form: $$\hat{G} = z_\pi^T A(s_0, z_d) z_\pi$$ where $A$ is a matrix-valued function of the initial state and dynamics embedding.

4. Evaluation and Fast Adaptation

In a new environment, the agent performs a few exploratory steps to infer the dynamics embedding $z_d$. Because the value function is quadratic in $z_\pi$, the optimal policy embedding $z_\pi^*$ can be found analytically using Singular Value Decomposition (SVD), bypassing the need for gradient-based optimization at test time.

Structure of the PD-VF value function and the optimization process.

Experimental Setup

The framework was evaluated on four continuous control domains with held-out test dynamics:

  • Spaceship: Navigating a 2D field of point charges.
  • MuJoCo Swimmer: Swimming against varying fluid currents.
  • MuJoCo Ant-Wind: Resisting external wind forces from arbitrary directions.
  • MuJoCo Ant-Legs: Walking with discrete variations in limb lengths.

Results and Discussion

PD-VF demonstrated near-instant adaptation, typically requiring only 1 to 4 steps of interaction to identify the environment's dynamics and solve for the optimal policy. It consistently outperformed meta-RL baselines like MAML and $RL^2$, which often require entire episodes or multiple gradient steps to adapt.

Key insights include:

  • Disentanglement: Separate encoders successfully decoupled "what the world does" from "what the agent does."
  • Latent Manifolds: t-SNE visualizations of $z_d$ revealed that the model learned the underlying structure of physical parameters (e.g., the circularity of wind direction) without explicit supervision.
  • Efficiency: The quadratic assumption turned a complex search problem into a linear algebra problem, enabling real-time adaptation.

Conclusion and Future Work

PD-VF provides a robust path toward agents that can adapt to physical changes almost instantly. Current limitations include the assumption of a fixed reward function and the use of low-dimensional state vectors. Future research will focus on scaling the framework to high-dimensional visual inputs and exploring more flexible value function architectures that maintain the speed of the quadratic approach.

Summary updated

BibTeX
@inproceedings{raileanu2020fast,
  author = {Roberta Raileanu and Maxwell Goldstein and Arthur Szlam and Rob Fergus},
  title = {{Fast Adaptation via Policy-Dynamics Value Functions}},
  booktitle = {International Conference on Machine Learning},
  year = {2020},
}
Figure