Overview
STOVE (STructured Object-aware Video Evaluation) is a state-space model that learns to "see" objects and "understand" their physical interactions directly from raw video without any human labels. By combining a structured image model with a Graph Neural Network (GNN) for dynamics, it achieves long-term physical consistency—such as energy conservation—and enables reinforcement learning agents to plan their moves with 10x to 30x better sample efficiency than standard model-free methods.
The Challenge of Object-Aware World Models
When humans watch a video of billiard balls, we perceive distinct objects with positions and velocities that obey the laws of physics. For AI, extracting this "object-aware" world model from unstructured, noisy video is a massive challenge.
Existing unsupervised models often struggle with two main issues:
- Training Complexity: They frequently use "double recurrence" (looping over time and then over every object), which is computationally expensive and unstable to train.
- Physical Realism: Many models "hallucinate" or fail to maintain physical laws over time. Objects might overlap, teleport, or lose energy inexplicably.
STOVE aims to build a model that is both efficient to train and physically grounded enough to serve as a reliable simulator for planning.
The STOVE Architecture
STOVE is a compositional state-space model built on two primary modules:
- The Image Model (SuPAIR): Based on "Sum-Product Attend-Infer-Repeat," this module handles vision. It detects objects and describes them using latent variables for position, size, and appearance. It leverages Sum-Product Networks (SPNs) to make inference tractable.
- The Dynamics Model (GNN): This module handles physics. It treats objects as nodes in a graph and uses a Graph Neural Network to predict how they interact (e.g., collisions or gravitational pull).
The Secret Sauce: Unlike previous works that use a completely separate network for inference, STOVE reuses the learned dynamics model to guide the inference process. It reconciles the "vision" prediction (where the camera sees the ball) with the "physics" prediction (where the ball should be based on its last known velocity) by multiplying their distributions. This acts as a powerful regularizer, ensuring the model adheres to the laws of physics it has learned.
By forcing the model to reason explicitly about velocities and positions, the authors bake strong inductive biases into the architecture. The model learns to predict state, not just pixels.
Experimental Setup and Key Results
The model was evaluated on synthetic but physically complex environments, including bouncing billiards, gravitational interactions, and a reinforcement learning (RL) variant with an agent-controlled ball.
- Superior Prediction and Long-Term Stability: On the Billiards and Gravity tasks, STOVE significantly outperformed existing unsupervised models (like VRNN, SQAIR, and DDPAE) in pixel-wise accuracy and position tracking. Remarkably, STOVE can generate realistic rollouts for over 100,000 timesteps, whereas baselines often saw objects overlap or disappear within 100 frames.
- Energy Conservation: A standout insight is the use of kinetic energy to measure model quality. STOVE maintains near-constant energy over thousands of frames, proving it captures the underlying rules of the environment.
- Sample-Efficient RL: By using STOVE as a simulator for Monte-Carlo Tree Search (MCTS), an agent was trained to avoid collisions. This agent reached peak performance in just 130,000 samples. In contrast, PPO (a popular model-free algorithm) required millions of samples to reach the same level.
- Closing the Gap: STOVE performs so well that it nearly matches the performance of supervised baselines that were given ground-truth object positions during training.
Limitations and Future Directions
Currently, STOVE assumes a fixed number of objects throughout the video and uses simple rectangular bounding boxes, which are not ideal for highly irregular or deformable objects. Furthermore, while the physics are complex, the visual environments tested are relatively simple. Scaling this approach to "in-the-wild" video remains a significant hurdle. Future work may explore more flexible object masks (similar to MONet or GENESIS) and the integration of more sophisticated planning algorithms.
Summary updated