Spatio-Temporal Image Boundary Extrapolation

Apratim Bhattacharyya, Mateusz Malinowski, Mario FritzarXiv 2016 · arXiv

Abstract

This paper introduces the novel task of predicting future object boundaries in video sequences. By focusing on structural boundaries rather than full RGB frames, the proposed approach avoids common issues with blurriness in future frame prediction. We evaluate several deep learning architectures, including a novel multi-scale model, on both real-world video benchmarks and synthetic physics environments. Results demonstrate that these models can learn complex motion dynamics and "intuitive physics" directly from visual data without explicit object definitions.

Introduction

Predicting future states in a video—future frame prediction—is a fundamental challenge in computer vision. Standard approaches often struggle with blurriness because pixel-level loss functions like Mean Squared Error (MSE) lead models to average out uncertain future positions, resulting in unusable images.

This work proposes a shift in focus: instead of predicting every pixel, we predict the crisp structural boundaries of future, unobserved frames. This task requires the model to understand spatio-temporal dynamics and extrapolate how object boundaries evolve over time. By predicting binary or confidence boundaries, the model avoids the worst of MSE-induced blurriness, as structural abstraction is easier for networks to extrapolate cleanly.

Methodology

The problem is framed as a patch-based prediction task. Given a sequence of past boundary patches (32x32 pixels), the model predicts future patches. Two primary modes are explored:

  1. Sequence-to-sequence: Predicting a block of future frames simultaneously.
  2. Recursive: Predicting one step ahead and using that prediction as input for the next step.
Overview of the recursive patch-based boundary prediction framework.

Architectures

Four distinct architectures were evaluated to balance spatial resolution and temporal context:

  • RNN Encoder-Decoder: A standard sequence model using GRUs.
  • Convolutional RNN: Integrates CNN layers with GRUs to better capture spatial features.
  • Fully Convolutional: An end-to-end CNN that avoids the information bottleneck of traditional autoencoders.
  • Multi-Scale: A Laplacian pyramid-inspired architecture that processes inputs at multiple resolutions. This allows for a large receptive field to capture long-range dependencies without losing fine-grained detail.
Comparison of the primary architectures: (a) Convolutional RNN, (b) Fully-Convolutional, and the Multi-Scale approach (bottom).

Experiments and Datasets

The models were tested on two distinct types of data:

  • VSB100 (Real Data): A real-world video segmentation benchmark. Ground-truth boundaries were extracted using hierarchical segmentation to provide stable training targets.
  • Synthetic Billiard Worlds: A controlled environment featuring 1 to 3 balls colliding on a frictionless surface. This dataset tests the model's ability to learn deterministic physical laws purely from visual input.

Results

Real-World Performance

On the VSB100 dataset, the Multi-Scale architecture proved superior, particularly in recursive mode with spatial context. It achieved an F-measure of 0.439 at 1 time-step into the future, declining to 0.258 at 4 time-steps.

Learning Intuitive Physics

The models demonstrated a remarkable ability to learn collision dynamics without explicit physical modeling or hardcoded physics engines.

  • For a single ball, the Fully-Convolutional model achieved an F-measure of 0.994 at $t+1$ and maintained 0.709 at $t+20$.
  • In complex 3-ball scenarios, the Multi-Scale model reached an F-measure of 0.968 at $t+1$ and 0.700 at $t+20$.
  • Curriculum Learning: Training sequentially on 1-ball, then 2-ball, and finally 3-ball environments was essential for mastering multi-object interactions.

Discussion and Conclusion

While the models excel at deterministic motion, they face challenges with the highly unstructured motion found in complex real-world scenes. In these cases, recursive predictions eventually lose structural integrity. Additionally, very long-term rollouts (e.g., 100 steps) can lead to physical hallucinations where objects vanish or change direction abruptly.

Despite these limitations, boundary extrapolation provides a sharp, structurally-aware alternative to pixel-level prediction. Future work could integrate these boundary "expectations" as priors for more robust video segmentation and tracking systems.

Summary updated

Abstract

Boundary prediction in images as well as video has been a very active topic of research and organizing visual information into boundaries and segments is believed to be a corner stone of visual perception. While prior work has focused on predicting boundaries for observed frames, our work aims at predicting boundaries of future unobserved frames. This requires our model to learn about the fate of boundaries and extrapolate motion patterns. We experiment on established real-world video segmentation dataset, which provides a testbed for this new task. We show for the first time spatio-temporal boundary extrapolation in this challenging scenario. Furthermore, we show long-term prediction of boundaries in situations where the motion is governed by the laws of physics. We successfully predict boundaries in a billiard scenario without any assumptions of a strong parametric model or any object notion. We argue that our model has with minimalistic model assumptions derived a notion of 'intuitive physics' that can be applied to novel scenes.
BibTeX
@misc{bhattacharyya2016spatio,
  author = {Apratim Bhattacharyya and Mateusz Malinowski and Mario Fritz},
  title = {{Spatio-Temporal Image Boundary Extrapolation}},
  year = {2016},
  eprint = {1605.07363},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1605.07363},
}
Figure