TaylorSwiftNet: Taylor Driven Temporal Modeling for Swift Future Frame Prediction

Mohammad Saber Pourheydari, Emad Bahrami Rad, Mohsen Fayyaz, Gianpiero Francesca, Mehdi Noroozi, Jürgen Gall

Introduction

Predicting future video frames is a fundamental challenge in computer vision, often plagued by accumulating errors and rigid temporal constraints. Traditional models typically treat video as a sequence of discrete frames, predicting them one by one. In "TaylorSwiftNet: Taylor Driven Temporal Modeling for Swift Future Frame Prediction," Pourheydari et al. propose a paradigm shift: modeling video dynamics as a continuous mathematical function using the Taylor series.

The Discretization Trap

Standard video prediction approaches, such as Recurrent Neural Networks (RNNs) like ConvLSTMs, are auto-regressive. They predict frame $t+1$ and use it as input for $t+2$. This sequential process leads to "error drift," where minor inaccuracies in early predictions snowball into significant blurriness over time. While some recent work uses Partial Differential Equations (PDEs) to model continuous motion, these are often discretized for computation, reintroducing step-by-step bottlenecks.

Methodology: TaylorSwiftNet Architecture

TaylorSwiftNet sidesteps these issues by approximating the motion function directly in a latent embedding space using a Taylor series expansion. Unlike "point estimators" that provide one specific frame, TaylorSwiftNet is a "function estimator" that learns the underlying rule of motion.

Overview of the TaylorSwiftNet architecture, showing the transition from discrete input frames to a continuous temporal model capable of parallel decoding at arbitrary time steps.

The architecture consists of three main components:

  1. Encoder: A modified 3DResNet maps a sequence of observed frames into a latent space $\mathcal{H}_t$.
  2. Delta Convolutional (DC) Blocks: These blocks recursively estimate the $n$-th order derivatives of the motion function at the current time.
  3. Taylor Expansion: The model constructs an analytical function $f(t + \tau)$ using the estimated derivatives. By treating motion as a function of the time offset $\tau$, the model can "jump" to any future point without calculating intermediate frames.
  4. Decoder: A 2D convolutional network maps the predicted latent state back into the image domain.

Key Advantages

  • Parallel Prediction: Unlike sequential RNNs, TaylorSwiftNet can predict multiple future frames simultaneously in a single forward pass.
  • Temporal Super-Resolution: Because $\tau$ is a continuous variable, the model can generate frames at higher frequencies than the training data (e.g., predicting at 30 FPS from 10 FPS data) by simply adjusting the input $\tau$.
  • Sampling Rate Independence: The model focuses on derivatives of change, making it robust to variations in input frame rates.

Experimental Results

The model was evaluated on diverse datasets including Moving MNIST, Human 3.6M (human walking), Traffic BJ (crowd flux), and Sea Surface Temperature (fluid dynamics).

  • Accuracy: On Moving MNIST, TaylorSwiftNet achieved a Mean Squared Error (MSE) of 17.8, significantly outperforming PhyDNet (24.4) and ConvLSTM (103.3).
  • Efficiency: In the Human 3.6M dataset, it achieved a runtime of 21ms per sequence, faster than PhyDNet (30ms) with fewer computational requirements (GMACs).
  • Long-term Stability: The model maintained higher structural similarity (SSIM) in long-term forecasting (up to 70 frames) compared to auto-regressive models that quickly lose detail.

Conclusion and Future Work

TaylorSwiftNet demonstrates that continuous temporal modeling via Taylor series offers a powerful alternative to discrete recurrent models. While it significantly reduces error drift and enables parallel inference, future work may explore adversarial losses to further reduce blurriness at long horizons and scaling the architecture to higher-resolution video.

Summary updated

Abstract

While recurrent neural networks (RNNs) demonstrate outstanding capabilities for future video frame prediction, they model dynamics in a discrete time space, i.e., they predict the frames sequentially with a fixed temporal step. RNNs are therefore prone to accumulate the error as the number of future frames increases. In contrast, partial differential equations (PDEs) model physical phenomena like dynamics in a continuous time space. However, the estimated PDE for frame forecasting needs to be numerically solved, which is done by discretization of the PDE and diminishes most of the advantages compared to discrete models. In this work, we, therefore, propose to approximate the motion in a video by a continuous function using the Taylor series. To this end, we introduce TaylorSwiftNet, a novel convolutional neural network that learns to estimate the higher order terms of the Taylor series for a given input video. TaylorSwiftNet can swiftly predict future frames in parallel and it allows to change the temporal resolution of the forecast frames on-the-fly. The experimental results on various datasets demonstrate the superiority of our model.
BibTeX
@inproceedings{pourheydari2022taylorswiftnet,
  author = {Mohammad Saber Pourheydari and Emad Bahrami Rad and Mohsen Fayyaz and Gianpiero Francesca and Mehdi Noroozi and Jürgen Gall},
  title = {{TaylorSwiftNet: Taylor Driven Temporal Modeling for Swift Future Frame Prediction}},
  booktitle = {BMVC},
  year = {2022},
}
Figure