Spatiotemporal Initialization for 3D CNNs with Generated Motion Patterns

Hirokatsu Kataoka, Kensho Hara, Ryusuke Hayashi, Eisuke Yamagata, Nakamasa Inoue

Abstract

Training 3D Convolutional Neural Networks (CNNs) for video recognition typically requires massive human-labeled datasets or "inflating" 2D weights. This paper introduces Video Perlin Noise (VPN) initialization, a Formula-Driven Supervised Learning (FDSL) approach that uses synthetic motion patterns to pre-train 3D kernels. VPN initialization consistently outperforms standard baselines, achieving up to a 10.3-point accuracy boost on ActivityNet.

Introduction: The Challenge of 3D Video Recognition

Training 3D CNNs is notoriously difficult due to the "parameter explosion" caused by the temporal dimension. These models are data-hungry and prone to overfitting. The current standard, I3D (Inflated 3D) initialization, copies ImageNet-trained 2D weights across time steps. However, this method fails to teach the model about motion during the initialization phase. This research proposes a dedicated spatiotemporal initialization mimicking biological visual processing.

Methods: Formula-Driven Supervised Learning

The authors utilize Formula-Driven Supervised Learning (FDSL) to generate synthetic motion patterns using Perlin noise.

Overview of the VPN initialization pipeline and its impact on early-layer filters.

The framework follows a three-stage process:

  1. VPN Initialization: 3D CNNs are trained on synthetic 3D blocks of noise. Labels are derived from the frequency parameters in the $x, y,$ and $t$ (time) dimensions. This pretext task forces kernels to develop basic spatiotemporal filters sensitive to orientation and direction.
  2. Kinetics Pre-training: The VPN-initialized model is further trained on large-scale natural video datasets like Kinetics-400.
  3. Fine-tuning: Finally, the model is fine-tuned on target benchmarks such as UCF-101, HMDB-51, or ActivityNet.

Datasets

  • Video Perlin Noise (VPN): A synthetic dataset of dynamic random textures defined by spatial and temporal frequencies.
  • Kinetics-400/700: Large-scale natural video datasets for intermediate pre-training.
  • UCF-101, HMDB-51, and ActivityNet: Standard benchmarks for final action recognition evaluation.

Key Insights

  • Bio-Mimicry: Training on Perlin noise encourages the development of "receptive fields" similar to localized oriented bandpass filters in the human visual cortex (area V1).
  • Frequency Sensitivity: Initializing with low-spatial frequency patterns yields superior results, suggesting that models benefit more from global shapes and slow-moving patterns than fine, noisy details.
  • Efficiency and Ethics: FDSL requires no real video for the initial phase, bypassing issues related to privacy, offensive content, and manual annotation errors.

Key Results

VPN initialization consistently outperforms standard I3D and scratch baselines across multiple architectures.

Performance comparison between scratch training and VPN initialization across different model depths.
  • ActivityNet: Achieved a massive 10.3 percentage point increase in top-1 accuracy over the baseline.
  • Kinetics-400: Improved top-1 accuracy by 0.9 pt over standard pre-training and 0.7 pt over I3D initialization.
  • UCF-101 & HMDB-51: Showed improvements of 4.8 pt and 6.2 pt respectively over scratch training.
  • Efficiency: The VPN dataset achieved these results with only 28,800 synthetic "videos," far fewer than the millions often required for self-supervised pre-training.

Limitations and Future Work

Current noise patterns are primarily axis-aligned. Future research will explore more complex synthetic textures, such as Gabor functions, to better simulate visual cortex properties. Additionally, combining FDSL with self-supervised methods could provide additive benefits for diverse video tasks.

Summary updated

Abstract

The paper proposes a framework of Formula-Driven Supervised Learning (FDSL) for spatiotemporal initialization. Our FDSL approach enables to automatically and simultaneously generate motion patterns and their video labels with a simple formula which is based on Perlin noise. We designed a dataset of generated motion patterns adequate for the 3D CNNs to learn a better basis set of natural videos. The constructed Video Perlin Noise (VPN) dataset can be applied to initialize a model before pre-training with large-scale video datasets such as Kinetics-400/700, to enhance target task performance. Our spatiotemporal initialization with VPN dataset (VPN initialization) outperforms the previous initialization method with the inflated 3D ConvNet (I3D) using 2D ImageNet dataset. Our proposed method increased the top-1 video-level accuracy of Kinetics-400 pre-trained model on {Kinetics-400, UCF-101, HMDB-51, ActivityNet} datasets. Especially, the proposed method increased the performance rate of Kinetics-400 pre-trained model by 10.3 pt on ActivityNet. We also report that the relative performance improvements from the baseline are greater in 3D CNNs rather than other models. Our VPN initialization mainly helps to enhance the performance in spatiotemporal 3D kernels. The datasets, codes and pre-trained models used in this study will be publicly available <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> .
BibTeX
@inproceedings{kataoka2022spatiotemporal,
  author = {Hirokatsu Kataoka and Kensho Hara and Ryusuke Hayashi and Eisuke Yamagata and Nakamasa Inoue},
  title = {{Spatiotemporal Initialization for 3D CNNs with Generated Motion Patterns}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2022},
  doi = {10.1109/wacv51458.2022.00081},
}
Figure