Abstract
This paper introduces the novel task of predicting future object boundaries in video sequences. By focusing on structural boundaries rather than full RGB frames, the proposed approach avoids common issues with blurriness in future frame prediction. We evaluate several deep learning architectures, including a novel multi-scale model, on both real-world video benchmarks and synthetic physics environments. Results demonstrate that these models can learn complex motion dynamics and "intuitive physics" directly from visual data without explicit object definitions.
Introduction
Predicting future states in a video—future frame prediction—is a fundamental challenge in computer vision. Standard approaches often struggle with blurriness because pixel-level loss functions like Mean Squared Error (MSE) lead models to average out uncertain future positions, resulting in unusable images.
This work proposes a shift in focus: instead of predicting every pixel, we predict the crisp structural boundaries of future, unobserved frames. This task requires the model to understand spatio-temporal dynamics and extrapolate how object boundaries evolve over time. By predicting binary or confidence boundaries, the model avoids the worst of MSE-induced blurriness, as structural abstraction is easier for networks to extrapolate cleanly.
Methodology
The problem is framed as a patch-based prediction task. Given a sequence of past boundary patches (32x32 pixels), the model predicts future patches. Two primary modes are explored:
- Sequence-to-sequence: Predicting a block of future frames simultaneously.
- Recursive: Predicting one step ahead and using that prediction as input for the next step.
Architectures
Four distinct architectures were evaluated to balance spatial resolution and temporal context:
- RNN Encoder-Decoder: A standard sequence model using GRUs.
- Convolutional RNN: Integrates CNN layers with GRUs to better capture spatial features.
- Fully Convolutional: An end-to-end CNN that avoids the information bottleneck of traditional autoencoders.
- Multi-Scale: A Laplacian pyramid-inspired architecture that processes inputs at multiple resolutions. This allows for a large receptive field to capture long-range dependencies without losing fine-grained detail.
Experiments and Datasets
The models were tested on two distinct types of data:
- VSB100 (Real Data): A real-world video segmentation benchmark. Ground-truth boundaries were extracted using hierarchical segmentation to provide stable training targets.
- Synthetic Billiard Worlds: A controlled environment featuring 1 to 3 balls colliding on a frictionless surface. This dataset tests the model's ability to learn deterministic physical laws purely from visual input.
Results
Real-World Performance
On the VSB100 dataset, the Multi-Scale architecture proved superior, particularly in recursive mode with spatial context. It achieved an F-measure of 0.439 at 1 time-step into the future, declining to 0.258 at 4 time-steps.
Learning Intuitive Physics
The models demonstrated a remarkable ability to learn collision dynamics without explicit physical modeling or hardcoded physics engines.
- For a single ball, the Fully-Convolutional model achieved an F-measure of 0.994 at $t+1$ and maintained 0.709 at $t+20$.
- In complex 3-ball scenarios, the Multi-Scale model reached an F-measure of 0.968 at $t+1$ and 0.700 at $t+20$.
- Curriculum Learning: Training sequentially on 1-ball, then 2-ball, and finally 3-ball environments was essential for mastering multi-object interactions.
Discussion and Conclusion
While the models excel at deterministic motion, they face challenges with the highly unstructured motion found in complex real-world scenes. In these cases, recursive predictions eventually lose structural integrity. Additionally, very long-term rollouts (e.g., 100 steps) can lead to physical hallucinations where objects vanish or change direction abruptly.
Despite these limitations, boundary extrapolation provides a sharp, structurally-aware alternative to pixel-level prediction. Future work could integrate these boundary "expectations" as priors for more robust video segmentation and tracking systems.
Summary updated