Flow and Depth Assisted Video Prediction with Latent Transformer

Eliyas Suleyman, Paul Henderson, Eksan Firkat, Nicolas PugeaultarXiv 2025 · arXiv · OpenReview

Abstract

Video prediction is a fundamental task for various downstream applications, including robotics and world modeling. Although general video prediction models have achieved remarkable performance in standard scenarios, occlusion is still an inherent challenge in video prediction. We hypothesize that providing explicit information about motion (via point-flow) and geometric structure (via depth-maps) will enable video prediction models to perform better in situations with occlusion and the background motion. To investigate this, we present the first systematic study dedicated to occluded video prediction. We use a standard multi-object latent transformer architecture to predict future frames, but modify this to incorporate information from depth and point-flow. We evaluate this model in a controlled setting on both synthetic and real-world datasets with not only appearance-based metrics but also Wasserstein distances on object masks, which can effectively measure the motion distribution of the prediction. We find that when the prediction model is assisted with point flow and depth, it performs better in occluded scenarios and predicts more accurate background motion compared to models without the help of these modalities.

Introduction

Video prediction—the task of generating future frames based on a past sequence—is a cornerstone of autonomous systems and world modeling. However, two persistent challenges remain: occlusion and background motion. Most current models rely primarily on pixel-level information. When an object disappears behind an obstacle, models often "forget" its existence, leading to predictions where objects vanish or fail to reappear correctly. Furthermore, without understanding the 3D structure of a scene, models struggle to distinguish between independent object motion and camera ego-motion.

Methodology

The proposed framework builds upon the Stochastic Class-Attended Transformer (SCAT), an object-centric model. The architecture utilizes an Object Aware Auto-Encoder (OAAE) to map video frames into a compact latent representation. The core innovation lies in the integration of two additional modalities during the encoding phase to provide explicit geometric and motion cues.

The proposed architecture integrates point-flow and depth maps into a latent transformer framework for video prediction.

Multi-Modal Integration

  1. Point-Flow (via CoTracker): Unlike dense, pixel-wise optical flow, point tracking follows specific key points on objects. These trackers can estimate a point's location even when it is temporarily occluded, providing a persistent motion trajectory that aids in predicting reappearance.
  2. Depth Maps (via DepthAnything-V2): Monocular depth estimation provides the model with the geometric "skeleton" of the scene. This helps the model reason about spatial constraints and distinguish between object movement and camera ego-motion.

The researchers evaluated several variants: SCAT-P (Point-flow), SCAT-D (Depth), and SCAT-PD (both), alongside the baseline SCAT and the popular SimVP model.

Experimental Setup

The study utilized two primary datasets to test different aspects of the problem:

  • Kubric-Occlusion: A controlled, synthetic dataset designed to feature clear occlusion events where a moving object passes behind a stationary one.
  • KITTI: A real-world autonomous driving dataset used to evaluate performance against complex ego-motion and real-world dynamics in residential and road scenes.

Results and Discussion

The inclusion of motion and depth cues yielded significant improvements in physical accuracy, though it highlighted interesting trade-offs in model capacity.

Trajectories Over Pixels

A key finding is that sparse point tracking is more robust than dense optical flow for long-range prediction. By maintaining a "memory" of an object's path, the model can accurately "hallucinate" an object's reappearance after an occlusion. In Kubric-Occlusion tests, only the models integrated with point-flow (SCAT-P and SCAT-PD) correctly predicted the moving object's reappearance.

Qualitative comparison on Kubric-Occlusion (left) and KITTI (right) datasets, demonstrating the superior occlusion recovery of flow-assisted models.

The Capacity Trade-off

The authors observed that adding modalities is not a "free lunch." When the model's parameter count is held constant (e.g., 8M or 11M parameters), incorporating depth and flow can slightly degrade visual sharpness (PSNR/SSIM) because the model must allocate limited latent space to represent the extra information. However, this trade-off results in much higher physical accuracy.

Motion vs. Appearance

There is a critical distinction between a prediction that looks visually sharp and one that is physically grounded. While some baselines produced sharper images, they often failed to predict correct object trajectories. The proposed flow-assisted models were superior in motion-based metrics like Earth Mover’s Distance (EMD) and Optical Flow Difference (OFD). For instance, on the Kubric dataset, SCAT-P achieved an OFD of 0.0356, significantly outperforming the baseline SCAT (0.0423).

Conclusion

Integrating explicit motion and geometric cues into latent transformers significantly enhances video prediction in the face of occlusion and complex ego-motion. While current results show a trade-off between visual fidelity and motion accuracy at smaller model scales, the findings suggest that future work should focus on scaling latent spaces and refining multi-modal fusion strategies to achieve both high-resolution reconstruction and physically accurate dynamics.

Summary updated

Abstract

Video prediction is a fundamental task for various downstream applications, including robotics and world modeling. Although general video prediction models have achieved remarkable performance in standard scenarios, occlusion is still an inherent challenge in video prediction. We hypothesize that providing explicit information about motion (via point-flow) and geometric structure (via depth-maps) will enable video prediction models to perform better in situations with occlusion and the background motion. To investigate this, we present the first systematic study dedicated to occluded video prediction. We use a standard multi-object latent transformer architecture to predict future frames, but modify this to incorporate information from depth and point-flow. We evaluate this model in a controlled setting on both synthetic and real-world datasets with not only appearance-based metrics but also Wasserstein distances on object masks, which can effectively measure the motion distribution of the prediction. We find that when the prediction model is assisted with point flow and depth, it performs better in occluded scenarios and predicts more accurate background motion compared to models without the help of these modalities.
BibTeX
@misc{suleyman2025flow,
  author = {Eliyas Suleyman and Paul Henderson and Eksan Firkat and Nicolas Pugeault},
  title = {{Flow and Depth Assisted Video Prediction with Latent Transformer}},
  year = {2025},
  eprint = {2511.16484},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2511.16484},
}
Figure