Abstract
Video prediction is a challenging task due to high dimensionality and inherent stochasticity. This paper proposes a grid-based keypoint learning framework that discretizes keypoint locations to improve accuracy and efficiency. By utilizing a novel condensation loss and representing keypoints as 2D binary maps, the model preserves spatial structures over long horizons. Experimental results on KTH, Human3.6M, and JIGSAWS datasets demonstrate that the proposed method achieves state-of-the-art performance while requiring 98% fewer computing resources than leading image-based models.
Introduction
Predicting future video frames from past observations is a fundamental challenge in computer vision. Existing approaches typically fall into two categories: image-based models, which predict pixels directly, and keypoint-based models, which track landmarks to warp reference frames. While image-based models are accurate in the short term, they are computationally expensive and prone to blurriness. Conversely, traditional keypoint models using continuous coordinates often suffer from "jitter" and compounding errors that distort human shapes in long-term predictions.
Methodology
The proposed framework consists of a three-stage pipeline: Grid Keypoint Detection, Grid Keypoint Prediction, and Frame Generation.
- Grid Keypoint Learning: Keypoints are mapped to a discrete $H \times W$ grid rather than continuous coordinates. To maintain differentiability, a "pushing" operation is used during training to allow gradient flow back to the detector.
- Condensation Loss: A novel loss function maximizes the "peakiness" of detection heatmaps, ensuring robust and unambiguous keypoint identification by encouraging single clear maxima and low values elsewhere.
- 2D Binary Map Representation: Keypoints are represented as 2D binary maps, preserving geometric structures that are often lost in 1D vector representations.
- Stochastic Propagation: A Variational Recurrent Neural Network (VRNN) architecture handles future uncertainty by learning a latent distribution for sampling multiple possible trajectories.
Experimental Results
The model was evaluated on the KTH, Human3.6M, and JIGSAWS datasets. On the KTH benchmark, it achieved a Structural Similarity (SSIM) of 0.837, significantly outperforming image-based baselines like SAVP-VAE (0.806) and keypoint-based competitors like Struct-VRNN (0.766).
Key insights from the evaluation include:
- Long-Term Stability: Qualitative results show that the grid-keypoint approach maintains a coherent human shape even after 50 frames, whereas other models lose the subject or distort limbs.
- Extreme Efficiency: The architecture is highly optimized, utilizing only 2.0M parameters compared to 22.8M for SVG-LP. This leads to over 98% savings in computing resources (GFLOPs), making it viable for edge device deployment.
Conclusion
This work demonstrates that discretization acts as an effective regularizer for video prediction, preventing coordinate drift and improving spatial awareness. By bridging the gap between efficient keypoint tracking and high-fidelity image synthesis, this framework provides a scalable solution for long-term video forecasting. Future work will explore integrating skeleton data and handling scenes with variable object counts.
Summary updated