Abstract
Video prediction requires modeling both scene appearance and complex dynamics. Current state-of-the-art models often rely on large, opaque recurrent neural networks (RNNs) that are computationally expensive and lack interpretability. We present Local Frequency Domain Transformer Networks (LFDTN), a lightweight alternative that shifts video prediction into the frequency domain. By processing local image patches through a Local Fourier Transform (LFT), LFDTN transforms spatial shifts into phase changes, allowing for a disentangled prediction pipeline. Our model achieves state-of-the-art performance on benchmarks like Moving MNIST++ and real-world NGSIM traffic data with up to 1,000x fewer parameters than traditional baselines. Furthermore, the frequency-domain approach provides human-interpretable velocity maps, offering insights into the model's internal decision-making process.
Introduction
Predicting the next frame in a video sequence is a fundamental challenge in computer vision, necessitating an understanding of both scene composition and temporal dynamics. Most current state-of-the-art models rely on complex recurrent layers like LSTMs or GRUs. While effective, these models suffer from two major flaws: a lack of interpretability, acting as "black boxes" that weave together motion extraction and rendering, and extreme computational heaviness, often requiring millions of parameters.
Researchers from the University of Bonn aim to "disentangle" these tasks using a differentiable architecture grounded in classical signal processing. By shifting the problem into the frequency domain, the model can treat motion as a simple geometric transformation rather than a complex learning task.
Methodology
The core innovation is the Local Frequency Domain Transformer Network (LFDTN). Instead of global image processing, the model breaks the frame into overlapping local cells.
Local Fourier Transform (LFT)
The model applies a Fourier Transform to these local cells. In the frequency domain, a spatial shift (movement) corresponds to a simple phase change. This allows the network to represent motion as a phase-addition problem, which is significantly easier to learn than pixel-level transitions.
Disentangled Pipeline
The prediction process is separated into three distinct, sequential steps:
- Extracting Transformations: Calculating the phase difference between consecutive frames to identify local shifts.
- Projecting Motion: A small "Transform Model" (a compact CNN) predicts how these local shifts will evolve.
- Applying Transformations: The predicted phase shifts are added back to the current frame's frequency representation.
Reconstruction and Segmentation
A novel Modified Inverse LFT (m_iLFT) reconstruction method ensures that moving objects do not "break" when crossing the boundaries of local cells. Additionally, the model can be integrated into a Kalman-filter-inspired architecture for self-supervised motion segmentation, separating foreground and background without human-provided labels.
Experimental Results
The model was evaluated on three benchmarks: Moving MNIST++, Moving MNIST on STL backgrounds, and NGSIM real-world traffic footage.
Parameter Efficiency
LFDTN demonstrates extreme efficiency. On the Moving MNIST++ dataset, the model achieved an L1 error of 0.00589 using only 3,000 parameters. For comparison, the HPNET baseline used 15.8 million parameters and achieved a higher error of 0.00681. This 1,000x reduction in parameters is achieved by baking the laws of translation and occlusion into the architecture as an inductive bias.
Real-World Accuracy and Interpretability
On the NGSIM traffic dataset, the model outperformed all baselines in Structural Similarity (SSIM) and L1 metrics. Because the model works with phase shifts, its internal representation of motion can be visualized as a vector field of local velocities—making the model's "thoughts" visible to humans.
Conclusion
LFDTN provides a lightweight and fully interpretable alternative to traditional video prediction models. By leveraging the properties of the frequency domain, it achieves superior performance with a fraction of the parameters. Future work will focus on multi-scale approaches to capture varying motion granularities and extending the architecture to handle more complex occlusions and non-rigid deformations.
Summary updated