Learning Video Dynamics with Predictive Differentiable Rendering

Yujin Tang, Tian Zhou, Xin Lin, Cheng Tan, Yifan Hu, Rong Jin, SouYoung Jin, Liang SunarXiv 2026 · arXiv

TL;DR

Traditional video prediction models often produce blurry results because they operate in discrete pixel space using Mean Squared Error (MSE) loss. This paper introduces Predictive Differentiable Rendering (PDR), a framework that bridges discrete pixel prediction with continuous 2D Gaussian representations via a lightweight, plug-and-play adapter. Powered by an ultra-fast, custom CUDA renderer (predgsplat), PDR generates sharp, high-fidelity future frames across both video and non-RGB scientific datasets at real-time speeds.

What Problem Does This Paper Address?

Whether forecasting the weather, modeling traffic, or analyzing human motion, predicting future video frames requires capturing complex spatial structures and temporal dynamics. Most state-of-the-art deterministic models operate entirely within a discrete pixel grid and optimize using pixel-wise losses like MSE.

While simple, MSE forces models to average out all possible future variations to minimize error. The result? Over-smoothed, blurry predictions that lack sharp boundaries and fine-grained details—a problem that compounds drastically over long-term forecasts. While generative approaches like diffusion models can produce sharper images, they require expensive iterative sampling, making them too slow for real-time applications.

Methods

The authors introduce the PDR framework to unify discrete pixel space and continuous Gaussian space within a single end-to-end pipeline.

Fig. 2: An overview of the PDR framework. Coarse predictions from a pixel-space backbone are refined using a lightweight PredGS adapter that learns per-frame 2D Gaussian parameters. These are rendered via a fast, differentiable 2D Gaussian renderer predgsplat and fused with the original output to enhance spatial detail.

The approach consists of three core components:

  1. The PredGS Adapter: A lightweight, plug-and-play module that can be attached to any existing pixel-space video predictor (e.g., TAU or SimVP). It takes the spatiotemporal features from the backbone and uses Multi-Layer Perceptrons (MLPs) to regress the parameters of $N$ 2D Gaussians per frame: coordinates ($\mu$), scale ($s$), rotation ($\theta$), and channel-aware amplitude ($a$).
  2. The predgsplat Renderer: A custom, fully vectorized CUDA kernel designed for efficient 2D Gaussian rasterization. Unlike traditional 3D Gaussian Splatting that relies on explicit opacity blending, predgsplat uses an amplitude-based formulation. This allows it to seamlessly scale to arbitrary channel dimensions ($C$), making it applicable to non-RGB data. It also bypasses costly per-Gaussian affine transformations by directly offsetting a predefined normalized base grid.
  3. Hybrid Loss Function: To overcome the blurring tendencies of MSE, the entire framework is optimized using a combined $L_1$ and Structural Similarity Index Measure (SSIM) loss function.

Datasets

The authors evaluated PDR across four diverse real-world benchmarks spanning different resolutions, horizons, and modalities:

  • TaxiBJ: A spatiotemporal dataset modeling urban traffic flow (non-RGB).
  • WeatherBench: A multi-variable meteorological dataset used here for 2-meter temperature forecasting.
  • KTH: A standard human action video dataset.
  • Human3.6M: A complex dataset focused on human pose and motion analysis.

Key Insights and Ideas

  • The Synergy of Representation and Loss: One of the paper's most striking findings is that continuous representation alone isn't a silver bullet. When trained with standard MSE loss, PDR actually underperforms the baseline pixel model. The continuous Gaussian field must be paired with a perceptually aligned loss ($L_1$ + SSIM) to properly unlock its detail-preserving capabilities.
  • Generalizing Beyond RGB: By replacing standard alpha-blending with channel-aware amplitudes, the authors successfully generalized 2D Gaussian Splatting to scientific, multi-channel data like temperature and traffic grids.
  • Uniform Grid Initialization Matters: When mapping features to Gaussian coordinates, initializing the MLP biases to a uniform Cartesian grid yielded much more stable and consistent training convergence than random sampling or feature-driven methods (like Harris corners).

Key Results

PDR consistently outperformed existing deterministic baselines across all datasets while introducing negligible computational overhead.

Fig. 1: (a) Paradigm and Qualitative Comparison. PDR bridges discrete pixel space prediction with continuous Gaussian space modeling through lightweight PredGS. PDR produces sharper and more accurate predictions than the pixel-based baseline. (b) Quantitative Comparison. When integrated into TAU [31], our PDR substantially reduces MSE on TaxiBJ [53] and WeatherBench [27] and improves SSIM on KTH [28] and Human3.6M [27], introducing marginal increases in parameters and FLOPs. Parameter and FLOP measurements are computed using fvcore [5] and the FLOPs difference between TAU and PDR on KTH and Human3.6M is below 1G.
  • Accuracy Gains: When integrated into the TAU baseline, PDR reduced MSE by 9.7% on TaxiBJ and 7.8% on WeatherBench.
  • Visual Fidelity: On video datasets, PDR achieved a 29.0% reduction in LPIPS (a metric for perceptual quality) on KTH and a 21.2% reduction on Human3.6M, successfully preserving sharp limb boundaries and fine textures.
  • Blazing Fast Efficiency: Thanks to the optimized CUDA kernel, predgsplat achieves up to a 10× speedup over baseline 2D renderers, exceeding 1000 FPS at $256\times256$ resolution.
  • vs. Diffusion Models: PDR runs 33× faster than the strongest diffusion competitor (ARFree) on the KTH dataset while achieving superior PSNR and SSIM scores.

Limitations and Future Work

While PDR bridges the gap between speed and sharpness, the authors note an expected trade-off: iterative diffusion models still hold a slight edge in absolute perceptual sharpness (lower LPIPS scores) because deterministic single-pass predictors inherently sacrifice some fine texture to maintain speed.

Additionally, because the model predicts Gaussian parameters on a per-frame basis, there is a theoretical risk of temporal flickering. Although their Temporal Warping Error (TWE) analysis shows that PDR remains well-behaved and close to the ground-truth noise floor, explicitly enforcing temporal consistency across the predicted Gaussian parameters represents a promising direction for future research.

Summary updated

Abstract

How to accurately predict a high-fidelity future world? While the visual world is inherently continuous, existing deterministic video prediction models operate in discrete pixel space and are mainly optimized with pixel-wise mean squared error (MSE), which often leads to over-smoothed predictions and a lack of fine-grained visual details. To address these limitations, we propose Predictive Differentiable Rendering (PDR), a novel end-to-end video prediction paradigm that bridges the gap between discrete and continuous representations. Inspired by recent progress in 3D reconstruction with 3D Gaussian Splatting, we introduce PredGS, a lightweight and plug-and-play adapter based on 2D Gaussian representation, which could be seamlessly integrated with existing pixel space predictors, significantly improving spatial detail preservation with negligible computational overhead. Furthermore, we develop predgsplat, a CUDA-accelerated differentiable 2D Gaussian renderer supporting arbitrary channels. Each Gaussian is defined by 5 + C learnable parameters (position, scale, rotation, and C channel amplitudes) and achieves up to 10x faster rendering than the baseline. Optimized by a combined L1 and SSIM loss, PDR overcomes the inherent blurring tendencies of MSE Loss, significantly enhancing the prediction performance. Extensive experiments on diverse real-world benchmarks, including TaxiBJ, WeatherBench, KTH, and Human3.6M, demonstrate that PDR consistently surpasses existing methods, delivering superior detail preservation, visual fidelity, and predictive accuracy.
BibTeX
@misc{tang2026learning,
  author = {Yujin Tang and Tian Zhou and Xin Lin and Cheng Tan and Yifan Hu and Rong Jin and SouYoung Jin and Liang Sun},
  title = {{Learning Video Dynamics with Predictive Differentiable Rendering}},
  year = {2026},
  eprint = {2606.31050},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2606.31050},
}
Figure