Taylor Swift: Taylor Driven Temporal Modeling for Swift Future Frame Prediction

Mohammad Saber Pourheydari, Mohsen Fayyaz, Emad Bahrami Rad, Mehdi Noroozi, Juergen Gall

Introduction

Predicting future video frames is a fundamental challenge in computer vision, often plagued by accumulating errors and rigid temporal constraints. Traditional models typically treat video as a sequence of discrete frames, predicting them one by one. In "TaylorSwiftNet: Taylor Driven Temporal Modeling for Swift Future Frame Prediction," Pourheydari et al. propose a paradigm shift: modeling video dynamics as a continuous mathematical function using the Taylor series.

The Discretization Trap

Standard video prediction approaches, such as Recurrent Neural Networks (RNNs) like ConvLSTMs, are auto-regressive. They predict frame $t+1$ and use it as input for $t+2$. This sequential process leads to "error drift," where minor inaccuracies in early predictions snowball into significant blurriness over time. While some recent work uses Partial Differential Equations (PDEs) to model continuous motion, these are often discretized for computation, reintroducing step-by-step bottlenecks.

Methodology: TaylorSwiftNet Architecture

TaylorSwiftNet sidesteps these issues by approximating the motion function directly in a latent embedding space using a Taylor series expansion. Unlike "point estimators" that provide one specific frame, TaylorSwiftNet is a "function estimator" that learns the underlying rule of motion.

Overview of the TaylorSwiftNet architecture, showing the transition from discrete input frames to a continuous temporal model capable of parallel decoding at arbitrary time steps.

The architecture consists of three main components:

  1. Encoder: A modified 3DResNet maps a sequence of observed frames into a latent space $\mathcal{H}_t$.
  2. Delta Convolutional (DC) Blocks: These blocks recursively estimate the $n$-th order derivatives of the motion function at the current time.
  3. Taylor Expansion: The model constructs an analytical function $f(t + \tau)$ using the estimated derivatives. By treating motion as a function of the time offset $\tau$, the model can "jump" to any future point without calculating intermediate frames.
  4. Decoder: A 2D convolutional network maps the predicted latent state back into the image domain.

Key Advantages

  • Parallel Prediction: Unlike sequential RNNs, TaylorSwiftNet can predict multiple future frames simultaneously in a single forward pass.
  • Temporal Super-Resolution: Because $\tau$ is a continuous variable, the model can generate frames at higher frequencies than the training data (e.g., predicting at 30 FPS from 10 FPS data) by simply adjusting the input $\tau$.
  • Sampling Rate Independence: The model focuses on derivatives of change, making it robust to variations in input frame rates.

Experimental Results

The model was evaluated on diverse datasets including Moving MNIST, Human 3.6M (human walking), Traffic BJ (crowd flux), and Sea Surface Temperature (fluid dynamics).

  • Accuracy: On Moving MNIST, TaylorSwiftNet achieved a Mean Squared Error (MSE) of 17.8, significantly outperforming PhyDNet (24.4) and ConvLSTM (103.3).
  • Efficiency: In the Human 3.6M dataset, it achieved a runtime of 21ms per sequence, faster than PhyDNet (30ms) with fewer computational requirements (GMACs).
  • Long-term Stability: The model maintained higher structural similarity (SSIM) in long-term forecasting (up to 70 frames) compared to auto-regressive models that quickly lose detail.

Conclusion and Future Work

TaylorSwiftNet demonstrates that continuous temporal modeling via Taylor series offers a powerful alternative to discrete recurrent models. While it significantly reduces error drift and enables parallel inference, future work may explore adversarial losses to further reduce blurriness at long horizons and scaling the architecture to higher-resolution video.

Summary updated

Abstract

While recurrent neural networks (RNNs) demonstrate outstanding capabilities in future video frame prediction, they model dynamics in a discrete time space and sequentially go through all frames until the desired future temporal step is reached. RNNs are therefore prone to accumulate the error as the number of future frames increases. In contrast, partial differential equations (PDEs) model physical phenomena like dynamics in continuous time space, however, current PDE-based approaches discretize the PDEs using e.g., the forward Euler method. In this work, we therefore propose to approximate the motion in a video by a continuous function using the Taylor series. To this end, we introduce TayloSwiftNet, a novel convolutional neural network that learns to estimate the higher order terms of the Taylor series for a given input video. TayloSwiftNet can swiftly predict any desired future frame in just one forward pass and change the temporal resolution on-the-fly. The experimental results on various datasets demonstrate the superiority of our model.
BibTeX
@inproceedings{pourheydari2022taylor,
  author = {Mohammad Saber Pourheydari and Mohsen Fayyaz and Emad Bahrami Rad and Mehdi Noroozi and Juergen Gall},
  title = {{Taylor Swift: Taylor Driven Temporal Modeling for Swift Future Frame Prediction}},
  booktitle = {BMVC},
  year = {2022},
}
Figure