Structured Object-Aware Physics Prediction for Video Modeling and Planning

Jannik Kossen, Karl Stelzner, Marcel Hussing, Claas Voelcker, Kristian Kersting

Overview

STOVE (STructured Object-aware Video Evaluation) is a state-space model that learns to "see" objects and "understand" their physical interactions directly from raw video without any human labels. By combining a structured image model with a Graph Neural Network (GNN) for dynamics, it achieves long-term physical consistency—such as energy conservation—and enables reinforcement learning agents to plan their moves with 10x to 30x better sample efficiency than standard model-free methods.

The Challenge of Object-Aware World Models

When humans watch a video of billiard balls, we perceive distinct objects with positions and velocities that obey the laws of physics. For AI, extracting this "object-aware" world model from unstructured, noisy video is a massive challenge.

Existing unsupervised models often struggle with two main issues:

  1. Training Complexity: They frequently use "double recurrence" (looping over time and then over every object), which is computationally expensive and unstable to train.
  2. Physical Realism: Many models "hallucinate" or fail to maintain physical laws over time. Objects might overlap, teleport, or lose energy inexplicably.

STOVE aims to build a model that is both efficient to train and physically grounded enough to serve as a reliable simulator for planning.

The STOVE Architecture

STOVE is a compositional state-space model built on two primary modules:

  1. The Image Model (SuPAIR): Based on "Sum-Product Attend-Infer-Repeat," this module handles vision. It detects objects and describes them using latent variables for position, size, and appearance. It leverages Sum-Product Networks (SPNs) to make inference tractable.
  2. The Dynamics Model (GNN): This module handles physics. It treats objects as nodes in a graph and uses a Graph Neural Network to predict how they interact (e.g., collisions or gravitational pull).
STOVE Architecture

The Secret Sauce: Unlike previous works that use a completely separate network for inference, STOVE reuses the learned dynamics model to guide the inference process. It reconciles the "vision" prediction (where the camera sees the ball) with the "physics" prediction (where the ball should be based on its last known velocity) by multiplying their distributions. This acts as a powerful regularizer, ensuring the model adheres to the laws of physics it has learned.

Graphical Model and Latent State

By forcing the model to reason explicitly about velocities and positions, the authors bake strong inductive biases into the architecture. The model learns to predict state, not just pixels.

Experimental Setup and Key Results

The model was evaluated on synthetic but physically complex environments, including bouncing billiards, gravitational interactions, and a reinforcement learning (RL) variant with an agent-controlled ball.

  • Superior Prediction and Long-Term Stability: On the Billiards and Gravity tasks, STOVE significantly outperformed existing unsupervised models (like VRNN, SQAIR, and DDPAE) in pixel-wise accuracy and position tracking. Remarkably, STOVE can generate realistic rollouts for over 100,000 timesteps, whereas baselines often saw objects overlap or disappear within 100 frames.
  • Energy Conservation: A standout insight is the use of kinetic energy to measure model quality. STOVE maintains near-constant energy over thousands of frames, proving it captures the underlying rules of the environment.
  • Sample-Efficient RL: By using STOVE as a simulator for Monte-Carlo Tree Search (MCTS), an agent was trained to avoid collisions. This agent reached peak performance in just 130,000 samples. In contrast, PPO (a popular model-free algorithm) required millions of samples to reach the same level.
  • Closing the Gap: STOVE performs so well that it nearly matches the performance of supervised baselines that were given ground-truth object positions during training.

Limitations and Future Directions

Currently, STOVE assumes a fixed number of objects throughout the video and uses simple rectangular bounding boxes, which are not ideal for highly irregular or deformable objects. Furthermore, while the physics are complex, the visual environments tested are relatively simple. Scaling this approach to "in-the-wild" video remains a significant hurdle. Future work may explore more flexible object masks (similar to MONet or GENESIS) and the integration of more sophisticated planning algorithms.

Summary updated

Abstract

When humans observe a physical system, they can easily locate components, understand their interactions, and anticipate future behavior, even in settings with complicated and previously unseen interactions. For computers, however, learning such models from videos in an unsupervised fashion is an unsolved research problem. In this paper, we present STOVE, a novel state-space model for videos, which explicitly reasons about objects and their positions, velocities, and interactions. It is constructed by combining an image model and a dynamics model in compositional manner and improves on previous work by reusing the dynamics model for inference, accelerating and regularizing training. STOVE predicts videos with convincing physical behavior over hundreds of timesteps, outperforms previous unsupervised models, and even approaches the performance of supervised baselines. We further demonstrate the strength of our model as a simulator for sample efficient model-based control, in a task with heavily interacting objects.
BibTeX
@inproceedings{kossen2020structured,
  author = {Jannik Kossen and Karl Stelzner and Marcel Hussing and Claas Voelcker and Kristian Kersting},
  title = {{Structured Object-Aware Physics Prediction for Video Modeling and Planning}},
  booktitle = {International Conference on Learning Representations},
  year = {2020},
}
Figure