Abstract
Training 3D Convolutional Neural Networks (CNNs) for video recognition typically requires massive human-labeled datasets or "inflating" 2D weights. This paper introduces Video Perlin Noise (VPN) initialization, a Formula-Driven Supervised Learning (FDSL) approach that uses synthetic motion patterns to pre-train 3D kernels. VPN initialization consistently outperforms standard baselines, achieving up to a 10.3-point accuracy boost on ActivityNet.
Introduction: The Challenge of 3D Video Recognition
Training 3D CNNs is notoriously difficult due to the "parameter explosion" caused by the temporal dimension. These models are data-hungry and prone to overfitting. The current standard, I3D (Inflated 3D) initialization, copies ImageNet-trained 2D weights across time steps. However, this method fails to teach the model about motion during the initialization phase. This research proposes a dedicated spatiotemporal initialization mimicking biological visual processing.
Methods: Formula-Driven Supervised Learning
The authors utilize Formula-Driven Supervised Learning (FDSL) to generate synthetic motion patterns using Perlin noise.
The framework follows a three-stage process:
- VPN Initialization: 3D CNNs are trained on synthetic 3D blocks of noise. Labels are derived from the frequency parameters in the $x, y,$ and $t$ (time) dimensions. This pretext task forces kernels to develop basic spatiotemporal filters sensitive to orientation and direction.
- Kinetics Pre-training: The VPN-initialized model is further trained on large-scale natural video datasets like Kinetics-400.
- Fine-tuning: Finally, the model is fine-tuned on target benchmarks such as UCF-101, HMDB-51, or ActivityNet.
Datasets
- Video Perlin Noise (VPN): A synthetic dataset of dynamic random textures defined by spatial and temporal frequencies.
- Kinetics-400/700: Large-scale natural video datasets for intermediate pre-training.
- UCF-101, HMDB-51, and ActivityNet: Standard benchmarks for final action recognition evaluation.
Key Insights
- Bio-Mimicry: Training on Perlin noise encourages the development of "receptive fields" similar to localized oriented bandpass filters in the human visual cortex (area V1).
- Frequency Sensitivity: Initializing with low-spatial frequency patterns yields superior results, suggesting that models benefit more from global shapes and slow-moving patterns than fine, noisy details.
- Efficiency and Ethics: FDSL requires no real video for the initial phase, bypassing issues related to privacy, offensive content, and manual annotation errors.
Key Results
VPN initialization consistently outperforms standard I3D and scratch baselines across multiple architectures.
- ActivityNet: Achieved a massive 10.3 percentage point increase in top-1 accuracy over the baseline.
- Kinetics-400: Improved top-1 accuracy by 0.9 pt over standard pre-training and 0.7 pt over I3D initialization.
- UCF-101 & HMDB-51: Showed improvements of 4.8 pt and 6.2 pt respectively over scratch training.
- Efficiency: The VPN dataset achieved these results with only 28,800 synthetic "videos," far fewer than the millions often required for self-supervised pre-training.
Limitations and Future Work
Current noise patterns are primarily axis-aligned. Future research will explore more complex synthetic textures, such as Gabor functions, to better simulate visual cortex properties. Additionally, combining FDSL with self-supervised methods could provide additive benefits for diverse video tasks.
Summary updated