Long-Term Image Boundary Prediction

Apratim Bhattacharyya, Mateusz Malinowski, Bernt Schiele, Mario Fritz

Abstract

Future frame prediction in video often suffers from significant blurriness due to inherent uncertainty in temporal evolution. This paper proposes a shift from predicting full RGB pixels to predicting image boundaries. By utilizing a novel Convolutional Multi-Scale Context (CMSC) architecture, the authors demonstrate that boundary prediction effectively captures essential scene dynamics and "intuitive physics" over long horizons. These predicted boundaries can further be used to sharpen standard RGB video predictions, significantly improving visual clarity and structural accuracy.

Introduction

Predicting the future is a hallmark of intelligence. In computer vision, this usually takes the form of future frame prediction: given a sequence of video frames, the model must estimate subsequent frames. However, the current state-of-the-art in RGB prediction faces a major hurdle: blurriness. Because the future is inherently uncertain, models trained with standard losses (like Mean Squared Error) tend to average out all possible outcomes, resulting in blurry blobs rather than crisp objects. This is especially problematic for high-frequency information—the sharp edges that define an object's shape and position.

The authors argue that boundaries are a superior representation for learning scene dynamics. Boundaries are compact, capture the essential structure of objects, and provide a rigorous benchmark for whether a model understands "intuitive physics" (e.g., how a billiard ball bounces) or "agent-based motion" (e.g., how a person moves).

The CMSC Architecture

The authors introduce the Convolutional Multi-Scale Context (CMSC) model, designed to preserve spatial information while capturing complex motion. Its design is driven by three key requirements:

  1. Fully Convolutional Architecture: To avoid the "information bottleneck" of fully connected layers, the model consists entirely of convolutional, pooling, and up-sampling layers, preserving spatial integrity throughout the network.
  2. Multi-Scale Prediction: Inspired by Laplacian pyramids, the model predicts boundaries at four different scales. Coarse predictions act as a guide for finer scales, allowing the model to handle both large-scale motion and fine-grained detail simultaneously.
  3. The Context Module: Instead of predicting a single image patch in isolation, the model processes a $3 \times 3$ grid of patches but only predicts the center one. This allows the model to "see" objects entering or leaving the patch, ensuring global consistency across the entire image and preventing edge artifacts.

For long-term prediction, the model operates recursively: it takes its own predicted boundary at time $t+1$ and feeds it back as input to predict the state at $t+2$.

Experimental Evaluation

The model was evaluated on a diverse set of challenges, ranging from deterministic synthetic billiards to complex natural video datasets like VSB100 and UCF101, which contain human and animal motions.

Key Insights and Results

  • Boundaries as a Physics Test: In RGB prediction, a model can "cheat" by blurring a moving object. In boundary prediction, if the edge is misplaced, the error is immediately obvious. This forces the model to learn underlying dynamics. On synthetic billiard tasks, the CMSC model maintained high accuracy 20 steps into the future (F-measure of 0.632), while simple baselines failed.
  • Handling Complexity: The model successfully learned ball-to-ball collisions in 6-ball scenarios, demonstrating an ability to model complex interactions rather than just linear motion.
  • Fusion for Sharpening: One of the most practical applications is using predicted boundaries to "sharpen" blurry RGB predictions. On the VSB100 dataset, this fusion approach improved the Peak Signal-to-Noise Ratio (PSNR) from 20.4 to 22.3 at four frames into the future.

Conclusion

Boundary prediction provides a "purer" test of a model's understanding of dynamics compared to traditional RGB prediction. By focusing on the structural essence of a scene, the CMSC architecture avoids the common pitfall of blurry predictions and provides a robust framework for long-term forecasting. Future work involves integrating this approach with generative models to better navigate the inherent non-determinism of human intent over extended periods.

Summary updated

Abstract

Boundary estimation in images and videos has been a very active topic of research, and organizing visual information into boundaries and segments is believed to be a corner stone of visual perception. While prior work has focused on estimating boundaries for observed frames, our work aims at predicting boundaries of future unobserved frames. This requires our model to learn about the fate of boundaries and corresponding motion patterns -- including a notion of "intuitive physics". We experiment on natural video sequences along with synthetic sequences with deterministic physics-based and agent-based motions. While not being our primary goal, we also show that fusion of RGB and boundary prediction leads to improved RGB predictions.
BibTeX
@inproceedings{bhattacharyya2018long,
  author = {Apratim Bhattacharyya and Mateusz Malinowski and Bernt Schiele and Mario Fritz},
  title = {{Long-Term Image Boundary Prediction}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2018},
  doi = {10.1609/aaai.v32i1.11811},
}
Figure