Abstract
While model-based deep reinforcement learning (RL) holds great promise for sample efficiency and generalization, learning an accurate dynamics model is often challenging and requires substantial interaction with the environment. A wide variety of domains have dynamics that share common foundations like the laws of classical mechanics, which are rarely exploited by existing algorithms. In this work, we propose an approach to learn task-agnostic dynamics priors from videos and incorporate them into an RL agent. Our method involves pre-training a frame predictor on task-agnostic physics videos to initialize dynamics models (and fine-tune them) for unseen target environments. Our frame prediction architecture, SpatialNet, is designed specifically to capture localized physical phenomena and interactions. Our approach allows for both faster policy learning and convergence to better policies, outperforming competitive approaches on several different environments. We also demonstrate that incorporating this prior allows for more effective transfer between environments.
Introduction
Reinforcement Learning (RL) is notoriously sample-inefficient, often requiring millions of interactions to learn simple tasks because agents typically start as a "blank slate." In contrast, humans possess strong dynamics priors—an intuitive sense of physics that allows us to predict how objects will move and interact regardless of the specific task at hand. Existing model-based RL attempts to learn environment dynamics, but these models are usually action-conditional and task-specific, leading to poor generalization. This work aims to bridge this gap by learning general, task-independent physical properties from raw video before an agent begins its specific RL task.
Methodology: SpatialNet and Task-Agnostic Pre-training
The proposed approach consists of two phases: unsupervised pre-training and RL with priors. First, a frame predictor is trained on task-agnostic videos to predict the next frame from a sequence of previous frames. This allows the model to learn universal physical properties like velocity and collisions without knowledge of specific agent actions.
The core of this method is SpatialNet, a novel architecture designed to preserve spatial information. Unlike standard RNNs that compress scenes into a single vector, SpatialNet utilizes a spatial memory block that performs convolutions over a 3D state representation. This inductive bias allows the network to track localized interactions—such as a ball hitting a corner—without losing track of the broader scene, preventing common failures like "object vanishing."
Experimental Evaluation
The authors evaluated their approach across several diverse domains:
- PhysVideos: A dataset of 5,000 trajectories of various shapes moving and colliding in 2D environments.
- PhysWorld: A suite of three 2D games: PhysGoal (navigation), PhysForage (collection), and PhysShooter (targeting).
- PhysShooter3D: A 3D environment testing understanding of parabolic trajectories and partial occlusions.
- Stochastic Atari: Ten games from the Arcade Learning Environment modified with "sticky actions" to test robustness.
Key Results
SpatialNet demonstrated superior multi-step forecasting, maintaining object shapes and environmental boundaries far better than ConvLSTM or RCNet baselines. When integrated into an RL agent (the Intuitive Physics Agent, or IPA), the physics prior led to significant performance gains. For instance, the IPA achieved 130% higher performance on PhysShooter and 56.5% higher scores on Atari Asteroids compared to competitive baselines.
Beyond final scores, the physics prior enabled faster convergence, allowing agents to earn rewards much earlier in training. The model also proved effective for transfer learning; a dynamics model fine-tuned on one task (e.g., PhysForage) could be transferred to another (PhysShooter), resulting in a 27% improvement over using the general physics prior alone.
Conclusion
By decoupling dynamics from policy and utilizing the SpatialNet architecture, this work demonstrates that task-agnostic physics priors can significantly enhance RL agent performance and efficiency. While challenges remain—such as domain mismatch when applying 2D priors to complex 3D tasks—the results suggest that learning an "intuitive sense of physics" is a powerful step toward more capable and adaptable artificial agents.
Summary updated