Local Frequency Domain Transformer Networks for Video Prediction

Hafez Farazi, Jan Nogga, Sven Behnke

Abstract

Video prediction requires modeling both scene appearance and complex dynamics. Current state-of-the-art models often rely on large, opaque recurrent neural networks (RNNs) that are computationally expensive and lack interpretability. We present Local Frequency Domain Transformer Networks (LFDTN), a lightweight alternative that shifts video prediction into the frequency domain. By processing local image patches through a Local Fourier Transform (LFT), LFDTN transforms spatial shifts into phase changes, allowing for a disentangled prediction pipeline. Our model achieves state-of-the-art performance on benchmarks like Moving MNIST++ and real-world NGSIM traffic data with up to 1,000x fewer parameters than traditional baselines. Furthermore, the frequency-domain approach provides human-interpretable velocity maps, offering insights into the model's internal decision-making process.

Introduction

Predicting the next frame in a video sequence is a fundamental challenge in computer vision, necessitating an understanding of both scene composition and temporal dynamics. Most current state-of-the-art models rely on complex recurrent layers like LSTMs or GRUs. While effective, these models suffer from two major flaws: a lack of interpretability, acting as "black boxes" that weave together motion extraction and rendering, and extreme computational heaviness, often requiring millions of parameters.

Researchers from the University of Bonn aim to "disentangle" these tasks using a differentiable architecture grounded in classical signal processing. By shifting the problem into the frequency domain, the model can treat motion as a simple geometric transformation rather than a complex learning task.

Methodology

The core innovation is the Local Frequency Domain Transformer Network (LFDTN). Instead of global image processing, the model breaks the frame into overlapping local cells.

Local Fourier Transform (LFT)

The model applies a Fourier Transform to these local cells. In the frequency domain, a spatial shift (movement) corresponds to a simple phase change. This allows the network to represent motion as a phase-addition problem, which is significantly easier to learn than pixel-level transitions.

Disentangled Pipeline

The prediction process is separated into three distinct, sequential steps:

  1. Extracting Transformations: Calculating the phase difference between consecutive frames to identify local shifts.
  2. Projecting Motion: A small "Transform Model" (a compact CNN) predicts how these local shifts will evolve.
  3. Applying Transformations: The predicted phase shifts are added back to the current frame's frequency representation.
LFDTN Architecture: The model utilizes a Local Fourier Transform (LFT) to process image patches, followed by a Transform Model that predicts future phase differences, which are then reconstructed using a modified inverse LFT (m_iLFT).

Reconstruction and Segmentation

A novel Modified Inverse LFT (m_iLFT) reconstruction method ensures that moving objects do not "break" when crossing the boundaries of local cells. Additionally, the model can be integrated into a Kalman-filter-inspired architecture for self-supervised motion segmentation, separating foreground and background without human-provided labels.

Experimental Results

The model was evaluated on three benchmarks: Moving MNIST++, Moving MNIST on STL backgrounds, and NGSIM real-world traffic footage.

Parameter Efficiency

LFDTN demonstrates extreme efficiency. On the Moving MNIST++ dataset, the model achieved an L1 error of 0.00589 using only 3,000 parameters. For comparison, the HPNET baseline used 15.8 million parameters and achieved a higher error of 0.00681. This 1,000x reduction in parameters is achieved by baking the laws of translation and occlusion into the architecture as an inductive bias.

Local Frequency Domain Transformations: Visualization of local patches and their corresponding phase differences, demonstrating how motion is captured in the frequency domain.

Real-World Accuracy and Interpretability

On the NGSIM traffic dataset, the model outperformed all baselines in Structural Similarity (SSIM) and L1 metrics. Because the model works with phase shifts, its internal representation of motion can be visualized as a vector field of local velocities—making the model's "thoughts" visible to humans.

Conclusion

LFDTN provides a lightweight and fully interpretable alternative to traditional video prediction models. By leveraging the properties of the frequency domain, it achieves superior performance with a fraction of the parameters. Future work will focus on multi-scale approaches to capture varying motion granularities and extending the architecture to handle more complex occlusions and non-rigid deformations.

Summary updated

Abstract

Video prediction is commonly referred to as forecasting future frames of a video sequence provided several past frames thereof. It remains a challenging domain as visual scenes evolve according to complex underlying dynamics, such as the camera's egocentric motion or the distinct motility per individual object viewed. These are mostly hidden from the observer and manifest as often highly non-linear transformations between consecutive video frames. Therefore, video prediction is of interest not only in anticipating visual changes in the real world but has, above all, emerged as an unsupervised learning rule targeting the formation and dynamics of the observed environment. Many of the deep learning-based state-of-the-art models for video prediction utilize some form of recurrent layers like Long Short-Term Memory (LSTMs) or Gated Recurrent Units (GRUs) at the core of their models. Although these models can predict the future frames, they rely entirely on these recurrent structures to simultaneously perform three distinct tasks: extracting transformations, projecting them into the future, and transforming the current frame. In order to completely interpret the formed internal representations, it is crucial to disentangle these tasks. This paper proposes a fully differentiable building block that can perform all of those tasks separately while maintaining interpretability. We derive the relevant theoretical foundations and showcase results on synthetic as well as real data. We demonstrate that our method is readily extended to perform motion segmentation and account for the scene's composition, and learns to produce reliable predictions in an entirely interpretable manner by only observing unlabeled video data.
BibTeX
@inproceedings{farazi2021local,
  author = {Hafez Farazi and Jan Nogga and Sven Behnke},
  title = {{Local Frequency Domain Transformer Networks for Video Prediction}},
  booktitle = {IJCNN},
  year = {2021},
  url = {https://openreview.net/forum?id=zJif56pRf-},
}
Figure