Spatiotemporal Joint Filter Decomposition in 3D Convolutional Neural Networks

Zichen Miao, Ze Wang, Xiuyuan Cheng, Qiang Qiu

Overview

Researchers have developed a novel way to decompose 3D convolutional filters into spatial "atoms," temporal "atoms," and joint coefficients. This approach, called Spatiotemporal Joint Filter Decomposition (STDCF), allows for an iterative training strategy that reduces GPU memory footprints by up to 50% while maintaining the high performance of 3D Convolutional Neural Networks (CNNs) on video tasks.

The Problem: The Memory Cost of 3D CNNs

3D CNNs are the gold standard for video understanding because they capture both spatial information (what is in the frame) and temporal information (how it moves) simultaneously. However, they are notorious memory hogs.

Because 3D filters are essentially cubes of weights, computational and memory costs explode as video resolution or clip length increases. To fit these models onto modern GPUs, researchers usually have to downsample videos—either shrinking the frames or skipping frames entirely. This creates a major bottleneck: if you downsample too much, you lose the fine-grained details or fast motions essential for accurate action recognition. The authors sought a way to model high-resolution, long-duration dependencies without needing a supercomputer's worth of VRAM.

Methods: Spatiotemporal Joint Filter Decomposition

The authors introduce STDCF to decouple spatial and temporal learning while preserving spatiotemporal dependency. Instead of a standard 3D filter, they decompose the weights into three distinct, learnable components:

  1. Spatial Atoms: A set of 2D filters that capture local textures and shapes.
  2. Temporal Atoms: A set of 1D filters that capture motion patterns over time.
  3. Joint Coefficients: A $1 \times 1$ convolutional layer that mixes the spatial and temporal atoms to reconstruct the spatiotemporal dependency.
Spatiotemporal Joint Filter Decomposition (STDCF) breaks a standard 3D filter into spatial atoms, temporal atoms, and joint coefficients.

This decomposition enables two key innovations:

  • Atom Swapping: Because the spatial and temporal components are separate, they can be "swapped" for versions with different dilations. This allows the model to handle different video resolutions or frame rates without retraining the entire filter. By simply dilating the temporal atoms, the model can maintain a consistent receptive field even when the input data's resolution changes.
  • Iterative Two-Stage Learning (ITSL): This is the core mechanism for memory saving. Instead of training on a full-resolution 3D video, the authors alternate between two stages:
  • Stage-T (Temporal Focus): Train on full-frame-rate videos that are spatially downsampled.
  • Stage-S (Spatial Focus): Train on full-resolution videos that are temporally downsampled.

By alternating these stages and using dilated atoms to bridge the resolution gap, the model learns the full spatiotemporal context using only half the memory.

Key Insights and Results

The authors evaluated their method on three major action recognition benchmarks: KTH, Kinetics-400, and Something-Something v1.

  • Beyond Rank-1 Decomposition: Previous efficient 3D models (like R(2+1)D) use rank-1 decomposition, which essentially treats space and time as completely independent. STDCF is more expressive; the joint coefficients allow it to capture complex correlations that rank-1 models miss, making it a "best of both worlds" between efficiency and power. In a toy "Translate-Rotate-MNIST" reconstruction task, STDCF significantly outperformed rank-1 decomposition (24.60 vs 21.50 PSNR).
  • Memory Efficiency: On the KTH dataset, STDCF reduced GPU memory usage by 50.17% compared to standard 3D CNNs.
  • Performance Parity: On Kinetics-400, an STDCF-ResNet50 model achieved 74.5% Top-1 accuracy. This is comparable to, and in some cases better than, state-of-the-art models like SlowFast or TSM, but with significantly fewer parameters (18.5M vs. 34.6M for SlowFast).

Limitations and Future Work

While the paper demonstrates significant memory savings, the Iterative Two-Stage Learning (ITSL) approach adds complexity to the training pipeline. It requires managing different data loading streams and alternating training phases, which might be more difficult to implement in standard deep learning frameworks than a simple end-to-end training run.

However, this decoupled learning paves the way for more accessible and scalable video understanding, potentially allowing researchers with limited hardware to train much deeper and more complex 3D models than previously possible. Future work could involve automating the atom selection or applying this decomposition to even more complex architectures like Video Transformers.

Summary updated

Abstract

In this paper, we introduce spatiotemporal joint filter decomposition to decouple spatial and temporal learning, while preserving spatiotemporal dependency in a video. A 3D convolutional filter is now jointly decomposed over a set of spatial and temporal filter atoms respectively. In this way, a 3D convolutional layer becomes three: a temporal atom layer, a spatial atom layer, and a joint coefficient layer, all three remaining convolutional. One obvious arithmetic manipulation allowed in our joint decomposition is to swap spatial or temporal atoms with a set of atoms that have the same number but different sizes, while keeping the remaining unchanged. For example, as shown later, we can now achieve tempo-invariance by simply dilating temporal atoms only. To illustrate this useful atom-swapping property, we further demonstrate how such a decomposition permits the direct learning of 3D CNNs with full-size videos through iterations of two consecutive sub-stages of learning: In the temporal stage, full-temporal downsampled-spatial data are used to learn temporal atoms and joint coefficients while fixing spatial atoms. In the spatial stage, full-spatial downsampled-temporal data are used for spatial atoms and joint coefficients while fixing temporal atoms. We show empirically on multiple action recognition datasets that, the decoupled spatiotemporal learning significantly reduces the model memory footprints, and allows deep 3D CNNs to model high-spatial long-temporal dependency with limited computational resources while delivering comparable performance.
BibTeX
@inproceedings{miao2021spatiotemporal,
  author = {Zichen Miao and Ze Wang and Xiuyuan Cheng and Qiang Qiu},
  title = {{Spatiotemporal Joint Filter Decomposition in 3D Convolutional Neural Networks}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=Alr5_kKmLBX},
}
Figure