Abstract
Future frame prediction in video often suffers from significant blurriness due to inherent uncertainty in temporal evolution. This paper proposes a shift from predicting full RGB pixels to predicting image boundaries. By utilizing a novel Convolutional Multi-Scale Context (CMSC) architecture, the authors demonstrate that boundary prediction effectively captures essential scene dynamics and "intuitive physics" over long horizons. These predicted boundaries can further be used to sharpen standard RGB video predictions, significantly improving visual clarity and structural accuracy.
Introduction
Predicting the future is a hallmark of intelligence. In computer vision, this usually takes the form of future frame prediction: given a sequence of video frames, the model must estimate subsequent frames. However, the current state-of-the-art in RGB prediction faces a major hurdle: blurriness. Because the future is inherently uncertain, models trained with standard losses (like Mean Squared Error) tend to average out all possible outcomes, resulting in blurry blobs rather than crisp objects. This is especially problematic for high-frequency information—the sharp edges that define an object's shape and position.
The authors argue that boundaries are a superior representation for learning scene dynamics. Boundaries are compact, capture the essential structure of objects, and provide a rigorous benchmark for whether a model understands "intuitive physics" (e.g., how a billiard ball bounces) or "agent-based motion" (e.g., how a person moves).
The CMSC Architecture
The authors introduce the Convolutional Multi-Scale Context (CMSC) model, designed to preserve spatial information while capturing complex motion. Its design is driven by three key requirements:
- Fully Convolutional Architecture: To avoid the "information bottleneck" of fully connected layers, the model consists entirely of convolutional, pooling, and up-sampling layers, preserving spatial integrity throughout the network.
- Multi-Scale Prediction: Inspired by Laplacian pyramids, the model predicts boundaries at four different scales. Coarse predictions act as a guide for finer scales, allowing the model to handle both large-scale motion and fine-grained detail simultaneously.
- The Context Module: Instead of predicting a single image patch in isolation, the model processes a $3 \times 3$ grid of patches but only predicts the center one. This allows the model to "see" objects entering or leaving the patch, ensuring global consistency across the entire image and preventing edge artifacts.
For long-term prediction, the model operates recursively: it takes its own predicted boundary at time $t+1$ and feeds it back as input to predict the state at $t+2$.
Experimental Evaluation
The model was evaluated on a diverse set of challenges, ranging from deterministic synthetic billiards to complex natural video datasets like VSB100 and UCF101, which contain human and animal motions.
Key Insights and Results
- Boundaries as a Physics Test: In RGB prediction, a model can "cheat" by blurring a moving object. In boundary prediction, if the edge is misplaced, the error is immediately obvious. This forces the model to learn underlying dynamics. On synthetic billiard tasks, the CMSC model maintained high accuracy 20 steps into the future (F-measure of 0.632), while simple baselines failed.
- Handling Complexity: The model successfully learned ball-to-ball collisions in 6-ball scenarios, demonstrating an ability to model complex interactions rather than just linear motion.
- Fusion for Sharpening: One of the most practical applications is using predicted boundaries to "sharpen" blurry RGB predictions. On the VSB100 dataset, this fusion approach improved the Peak Signal-to-Noise Ratio (PSNR) from 20.4 to 22.3 at four frames into the future.
Conclusion
Boundary prediction provides a "purer" test of a model's understanding of dynamics compared to traditional RGB prediction. By focusing on the structural essence of a scene, the CMSC architecture avoids the common pitfall of blurry predictions and provides a robust framework for long-term forecasting. Future work involves integrating this approach with generative models to better navigate the inherent non-determinism of human intent over extended periods.
Summary updated