Towards Robust Deterministic and Probabilistic Modeling for Predictive Learning

Xuesong Nie, Haoyuan Jin, Vijayakumar Bhagavatula, Xiaofeng LiuIJCAI 2025 · DOI

Introduction

Predicting future states in spatiotemporal sequences—ranging from video frames to traffic patterns—is a fundamental challenge in machine learning. The primary difficulty lies in the "entanglement" of visual dynamics. A scene typically consists of stable, deterministic elements (like background textures or clothing patterns) and highly uncertain, probabilistic motion (like a person's specific path or a car's turn). Existing models often fail to balance these, resulting in either blurry details or unrealistic motion artifacts.

The DDP Framework: Disentangling Motion and Visuals

To address these issues, researchers introduced the Disentangling Deterministic and Probabilistic modeling (DDP) framework. DDP explicitly separates the prediction task into two distinct pathways in the latent space:

1. Probabilistic Motion Modeling

Instead of relying on traditional one-to-one optical flow, DDP utilizes Motion Flow. This approach captures "many-to-many" relationships between patches across frames, allowing for more flexible modeling of complex interactions and occlusions. To handle the inherent randomness of future movement, a Motion Diffuser—powered by a spatiotemporal State Space Model (SSM)—iteratively refines motion predictions from Gaussian noise.

Comparison of implicit modeling via RNNs versus explicit modeling using Optical and Motion Flow.

2. Deterministic Visual Modeling

To preserve sharp visual details, the Spectral-Aware Module (SAM) operates in the frequency domain. By using Fourier Transforms, the model can identify and amplify features that should remain stable over time, preventing the "fading" or "blurring" common in spatial-only models. Finally, the model uses the predicted motion flow to "warp" these stable features from past frames into their predicted future positions.

Experimental Evaluation

The DDP framework was tested across five diverse datasets to prove its robustness across different data modalities: UCF Sports (human action), KITTI & Caltech (driving), Moving MNIST (synthetic), TaxiBJ (traffic flow), and WeatherBench (climate data).

Example of patch-based tracking in a real-world video sequence from the UCF Sports dataset.

The model consistently achieved state-of-the-art performance across all benchmarks. For instance, on the UCF Sports dataset, DDP reached a PSNR of 32.16, significantly outperforming previous benchmarks in visual clarity. In driving scenes, it maintained sharp lane lines and object boundaries where other models produced warped artifacts. Notably, its success on WeatherBench (achieving the lowest RMSE of 1.040) demonstrates that the DDP framework is robust enough to handle abstract spatiotemporal data beyond standard video.

Conclusion and Future Work

By disentangling "what things look like" from "where things go," DDP provides a more robust foundation for predictive learning. While computational costs associated with iterative diffusion processes remain a challenge for real-time scaling to 4K video or very long sequences, the framework's ability to maintain high-fidelity details alongside complex motion marks a significant step forward. Future work will likely focus on optimizing inference speed and memory management for real-world applications like autonomous driving.

Summary updated

BibTeX
@inproceedings{nie2025towards,
  author = {Xuesong Nie and Haoyuan Jin and Vijayakumar Bhagavatula and Xiaofeng Liu},
  title = {{Towards Robust Deterministic and Probabilistic Modeling for Predictive Learning}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.24963/ijcai.2025/195},
}
Figure