Abstract
Humans are masters at quickly learning many complex tasks, relying on an approximate understanding of the dynamics of their environments. In much the same way, we would like our learning agents to build internal models that allow for efficient planning and rapid adaptation. We introduce Imagined Value Gradients (IVG), a model-based reinforcement learning algorithm that learns to plan in a compressed, latent "imagination" space. By backpropagating gradients through imagined trajectories, IVG matches the performance of state-of-the-art model-free methods while significantly improving data efficiency and the ability to transfer learned physics to new robotic manipulation tasks.
Introduction
In reinforcement learning (RL), model-free algorithms learn policies through sheer trial and error. While successful, they are notoriously data-hungry and task-specific; an agent that learns to stack blocks cannot easily reuse that knowledge to push them.
Model-based RL attempts to solve this by having the agent learn a predictive model of the environment's dynamics—essentially an internal physics engine. In theory, this understanding of the world should be transferable to new tasks. However, learning dynamics directly from high-dimensional inputs like video feeds is difficult. Small errors in predicting future frames compound over time, making long-term planning unreliable. This paper introduces Imagined Value Gradients (IVG) to build a robust, vision-based dynamics model that accelerates learning and transfers reliably.
The IVG Framework
Instead of trying to predict future pixels, IVG uses an encoder to compress raw images and robot joint data into a low-dimensional "latent state." It then learns a deterministic transition model to predict how this latent state will change given a specific action.
To improve its policy, the agent "imagines" a sequence of actions $N$ steps into the future entirely within this latent space. Because the entire model (encoder, transition dynamics, reward predictor, and policy) is a differentiable neural network, the agent can use standard backpropagation through time to update its policy directly from these imagined trajectories. To prevent the compounding errors typical of long rollouts, IVG calculates the policy gradient for multiple rollout lengths (e.g., 1 to 5 steps) and averages them.
Experimental Setup
The experiments were conducted in simulated MuJoCo environments featuring a Sawyer robotic arm. Inputs included continuous 5D control, proprioception, and two 64x64 RGB camera views. The agent was tested on tasks such as lifting, stacking, and matching positions. The researchers also introduced environments with visual distractors (unseen objects) and noisy observations (delayed sensors, flickering colors) to test model robustness.
Key Results
IVG demonstrates significant improvements in both learning speed and final performance compared to strong model-free baselines like Maximum a Posteriori Policy Optimisation (MPO).
- Efficiency: On the complex "Stack" task, IVG learns about 2x faster than MPO and SVG(0).
- Transfer Learning: When transferring a multi-task pre-trained model to a new task with unseen visual distractors, IVG learns 3-4x faster than learning from scratch.
- Robustness: Training the dynamics model on multiple tasks forces the latent space to learn general physics rather than task-specific shortcuts. This makes the model resilient to new tasks and visual distractors, even unlocking sparse-reward tasks where model-free baselines fail.
Limitations and Future Work
While IVG is a step forward, it assumes deterministic latent dynamics. Highly stochastic environments might require probabilistic transition models. Additionally, the current setup relies on fixed external camera views; future work could extend this to egocentric cameras to better handle occlusions. Finally, while more data-efficient than model-free methods, IVG still requires thousands of episodes, meaning further improvements are needed before direct deployment on physical robotics hardware without simulation pre-training.
Summary updated