SyncVP: Joint Diffusion for Synchronous Multi-Modal Video Prediction

Enrico Pallotta, Sina Mokhtarzadeh Azar, Shuai Li, Olga Zatsarynna, Juergen Gall

Abstract

Synchronous multi-modal video prediction requires maintaining temporal consistency and semantic alignment across disparate data types, such as RGB frames, depth maps, and semantic labels. We present SyncVP, a novel framework based on Latent Diffusion Models that jointly predicts future frames for multiple modalities. By employing Split Spatio-Temporal Cross-Attention (STCA) and a shared noise schedule during forward diffusion, SyncVP achieves superior alignment and computational efficiency. Our experiments across driving, robotics, and climate datasets demonstrate that SyncVP significantly outperforms existing diffusion-based baselines in Fréchet Video Distance (FVD) while enabling robust "hallucination" of missing modalities during inference.

1. Introduction

Predicting future states in a video sequence is a fundamental challenge in computer vision, with applications ranging from autonomous driving to robotic manipulation. While standard video prediction models focus primarily on RGB pixels, real-world dynamics are often better captured through multi-modal data. However, integrating modalities like depth or semantic maps introduces significant hurdles: different data types possess distinct statistical properties, and maintaining perfect synchronization between them is computationally demanding.

Previous approaches often relied on unidirectional conditioning (e.g., using depth to predict RGB), which fails to capture the reciprocal relationship between modalities. SyncVP addresses these limitations by treating multi-modal prediction as a joint diffusion process, ensuring that all modalities evolve synchronously in the latent space.

2. Methodology

SyncVP builds upon the Latent Diffusion Model (LDM) framework, specifically the Projected Video Diffusion Model (PVDM) architecture. To bridge the multi-modal gap, the framework introduces three primary innovations:

2.1 Split Spatio-Temporal Cross-Attention (STCA)

To facilitate interaction between modalities without the prohibitive cost of global attention, SyncVP utilizes STCA. This mechanism factorizes cross-attention into separate spatial and temporal dimensions. By sharing high-level semantic features across modalities in this factorized manner, the model achieves a 37% to 50% reduction in computational overhead compared to vanilla cross-attention while maintaining high performance.

2.2 Shared Forward Diffusion

A core insight of SyncVP is the use of shared noise. During the training phase, the exact same Gaussian noise vector is added to both modalities. This forces the independent denoising branches to learn a shared reverse transformation. This shared target acts as a powerful regularizer, significantly improving loss convergence and ensuring that the generated modalities remain perfectly aligned in time and space.

2.3 Cross-Modality Guidance

To handle real-world scenarios where sensors might fail, SyncVP incorporates a training strategy inspired by classifier-free guidance. By randomly masking one input modality during training, the model learns to predict the future of both modalities even when only one is provided at inference time. This allows the model to "hallucinate" missing data, such as generating future depth maps from only past RGB frames.

3. Experimental Setup

The versatility of SyncVP was evaluated across five diverse datasets:

  • Cityscapes: Real-world driving scenes featuring RGB and disparity maps.
  • BAIR: Robotic arm interactions, with depth maps generated via DepthAnything-v2.
  • SYNTHIA: Synthetic urban environments for RGB and semantic segmentation prediction.
  • ERA5-Land: A climate dataset for predicting surface pressure and temperature.
  • OpenDV-YouTube: Large-scale driving videos used for high-resolution (256x256) validation.

4. Results and Discussion

SyncVP establishes a new state-of-the-art for multi-modal video prediction across several metrics:

  • Temporal Quality: On the Cityscapes and BAIR benchmarks, SyncVP improved the Fréchet Video Distance (FVD) by 21% and 27%, respectively, compared to previous diffusion models like STDiff.
  • Inference Efficiency: The model is significantly faster than its predecessors. For a 28-frame sequence at 128x128 resolution, SyncVP requires approximately 22.68 seconds on a single GPU, compared to 239 seconds for STDiff.
  • Cross-Domain Generalization: The framework's success on the ERA5-Land climate dataset demonstrates that SyncVP is not limited to visual modalities but can effectively synchronize any paired spatio-temporal data.

5. Conclusion and Limitations

SyncVP demonstrates that joint diffusion with shared noise and factorized attention is a highly effective strategy for synchronous multi-modal prediction. While the model excels in temporal consistency and perceptual quality (FVD), it shows slightly lower scores in frame-by-frame metrics like SSIM and LPIPS compared to some non-diffusion baselines. This is likely due to the high compression ratio of the latent autoencoder. Future research will focus on scaling the architecture to support more than two modalities and integrating advanced autoencoders to preserve finer spatial details for downstream tasks like path planning.

Summary updated

BibTeX
@inproceedings{pallotta2025syncvp,
  author = {Enrico Pallotta and Sina Mokhtarzadeh Azar and Shuai Li and Olga Zatsarynna and Juergen Gall},
  title = {{SyncVP: Joint Diffusion for Synchronous Multi-Modal Video Prediction}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2025},
  doi = {10.1109/cvpr52734.2025.01287},
}
Figure