Abstract
Standard reinforcement learning (RL) agents are often brittle, failing when environment dynamics—such as friction or gravity—deviate from training conditions. This paper proposes Policy-Dynamics Value Functions (PD-VF), a framework designed for rapid adaptation to novel dynamics. By conditioning a value function on latent embeddings of both agent policies and environment physics, PD-VF allows for the analytical derivation of optimal policies. Experiments across several continuous control tasks demonstrate that PD-VF can adapt to unseen environments in as few as one to four interaction steps, significantly outperforming existing meta-RL benchmarks.
Introduction
Reinforcement learning has achieved remarkable success in static environments, but real-world deployment requires agents to handle unpredictable changes in physics. For instance, a robot walking on grass must quickly adjust its gait when moving onto ice. Traditional adaptation methods often require extensive re-training or multiple gradient updates, which are too slow for time-critical applications. PD-VF addresses this by framing adaptation as a fast "policy calculation" rather than a slow optimization search.
Methodology
The PD-VF framework decomposes the adaptation problem into four distinct phases:
1. Reinforcement Learning Phase
A library of policies is generated by training an ensemble of agents using Proximal Policy Optimization (PPO) across various environments with diverse dynamics. Checkpoints are saved throughout training to capture a wide range of behaviors, from novice to expert.
2. Self-Supervised Embedding Phase
To represent policies and dynamics in a latent space, the authors employ Transformer-based encoders:
- Dynamics Encoder ($E_d$): Learns a representation $z_d$ by predicting the next state $s_{t+1}$ given the current state and action.
- Policy Encoder ($E_\pi$): Learns a representation $z_\pi$ by predicting the actions $a_t$ taken by a specific policy.
3. Supervised Training Phase
The core of the method is the training of a value function $W(s_0, z_\pi, z_d)$ that predicts the expected return $\hat{G}$. To enable fast optimization, $W$ is constrained to a quadratic form: $$\hat{G} = z_\pi^T A(s_0, z_d) z_\pi$$ where $A$ is a matrix-valued function of the initial state and dynamics embedding.
4. Evaluation and Fast Adaptation
In a new environment, the agent performs a few exploratory steps to infer the dynamics embedding $z_d$. Because the value function is quadratic in $z_\pi$, the optimal policy embedding $z_\pi^*$ can be found analytically using Singular Value Decomposition (SVD), bypassing the need for gradient-based optimization at test time.
Experimental Setup
The framework was evaluated on four continuous control domains with held-out test dynamics:
- Spaceship: Navigating a 2D field of point charges.
- MuJoCo Swimmer: Swimming against varying fluid currents.
- MuJoCo Ant-Wind: Resisting external wind forces from arbitrary directions.
- MuJoCo Ant-Legs: Walking with discrete variations in limb lengths.
Results and Discussion
PD-VF demonstrated near-instant adaptation, typically requiring only 1 to 4 steps of interaction to identify the environment's dynamics and solve for the optimal policy. It consistently outperformed meta-RL baselines like MAML and $RL^2$, which often require entire episodes or multiple gradient steps to adapt.
Key insights include:
- Disentanglement: Separate encoders successfully decoupled "what the world does" from "what the agent does."
- Latent Manifolds: t-SNE visualizations of $z_d$ revealed that the model learned the underlying structure of physical parameters (e.g., the circularity of wind direction) without explicit supervision.
- Efficiency: The quadratic assumption turned a complex search problem into a linear algebra problem, enabling real-time adaptation.
Conclusion and Future Work
PD-VF provides a robust path toward agents that can adapt to physical changes almost instantly. Current limitations include the assumption of a fixed reward function and the use of low-dimensional state vectors. Future research will focus on scaling the framework to high-dimensional visual inputs and exploring more flexible value function architectures that maintain the speed of the quadratic approach.
Summary updated