Abstract
In order to interact with the world, agents must be able to predict the results of the world's dynamics. A natural approach to learn about these dynamics is through video prediction, as cameras are ubiquitous and powerful sensors. Direct pixel-to-pixel video prediction is difficult, does not take advantage of known priors, and does not provide an easy interface to utilize the learned dynamics. Object-centric video prediction offers a solution to these problems by taking advantage of the simple prior that the world is made of objects and by providing a more natural interface for control. However, existing object-centric video prediction pipelines require dense object annotations in training video sequences. In this work, we present Object-centric Prediction without Annotation (OPA), an object-centric video prediction method that takes advantage of priors from powerful computer vision models. We validate our method on a dataset comprised of video sequences of stacked objects falling, and demonstrate how to adapt a perception model in an environment through end-to-end video prediction training.
Introduction
Predicting future states is a fundamental requirement for autonomous agents. Whether a robot is catching a ball or stacking blocks, it must simulate the consequences of its actions and the environment's physics. Traditional "pixel-to-pixel" models attempt to map current frames directly to future ones, a task that is computationally expensive and often results in blurry predictions as the model struggles to learn geometry, lighting, and physics simultaneously from raw pixels.
Object-centric models offer a more structured alternative by treating the world as a collection of discrete entities. However, these models typically rely on dense human-labeled annotations—such as bounding boxes or segmentation masks for every frame—which limits their scalability. This research introduces OPA (Object-centric Prediction without Annotation), a method that learns to segment and predict object dynamics using only raw, unlabeled video.
Methodology
The OPA pipeline integrates three core components into a self-supervised, end-to-end training loop.
1. Instance Segmentation
To identify objects without human labels, OPA leverages optical flow—the pattern of apparent motion between consecutive frames. By assuming that pixels moving together belong to the same object, the system generates "pseudo-ground truth" masks. These masks are used to fine-tune a Mask R-CNN segmenter, allowing the model to "discover" objects based on their motion.
2. Dynamics Prediction
Once objects are segmented, a dynamics module (a fully-connected neural network) processes the state of each object, including its image patch and bounding box. It predicts the object's future location and appearance in the subsequent frame.
3. Image Generation and Compositing
The final stage involves re-assembling the full image from predicted object states. This process uses a compositing step where the model predicts the background, the individual objects, and "synthetic pixels" to handle occlusions and fill gaps left by moving objects.
The entire system is optimized by minimizing the difference between the predicted frame and the actual next frame in the video sequence, creating a self-correcting feedback loop.
Experiments
The method was evaluated using an extension of the Shapestacks dataset, which consists of video sequences of unstable block towers falling. This dataset is a standard benchmark for testing physical intuition, including gravity, collisions, and friction.
Key Results
OPA was compared against CVP (a state-of-the-art supervised model) and No-factor (a non-object-centric baseline).
- Quantitative Performance: OPA achieved a Mean Square Error (MSE) of 0.0097, slightly outperforming the CVP model (0.0101) despite CVP having access to ground-truth bounding boxes during training.
- Qualitative Consistency: OPA maintained object integrity significantly better than non-object-centric models. While baselines often produced "blurry" or "melting" blocks over time, OPA's predictions remained sharp and physically plausible.
- Perception Adaptation: A key finding was that the perception model (the segmenter) improved during training. By being forced to provide useful features for the dynamics brain, the segmenter learned to recognize objects even when they were static, despite initially relying on motion cues.
Discussion and Limitations
While OPA demonstrates the power of motion-based self-supervision, it has limitations. The model requires objects to move at some point during training to be "discovered." Additionally, the current tracking mechanism relies on simple centroid proximity, which may struggle in highly crowded scenes with fast-moving, overlapping objects. Future work will focus on testing OPA in more complex, "in-the-wild" environments with diverse textures and lighting conditions.
Conclusion
OPA demonstrates that object-centric video prediction is possible without expensive manual annotations. By leveraging optical flow as a self-supervisory signal, agents can learn to perceive and predict the dynamics of their environment in a scalable, data-driven manner.
Summary updated