Abstract
Manual design of spatio-temporal neural networks for video action recognition is a complex, resource-intensive process. This paper presents the first application of Neural Architecture Search (NAS) to this domain, utilizing a differentiable search space and efficient "pseudo-3D" operators. The resulting algorithmically discovered model is significantly more efficient than human-designed counterparts, achieving 58.6% accuracy on the UCF101 dataset with only 0.67 million parameters—approximately 1% of the size of standard 3D-ConvNets.
Introduction
Action recognition requires capturing both spatial context and temporal dynamics. While 3D Convolutional Neural Networks (3D-CNNs) are the standard approach, their manual design is a grueling process of trial and error. Furthermore, full 3D convolutions are computationally expensive and prone to overfitting. Although NAS has revolutionized 2D image tasks, its application to 3D video data has been limited by the "curse of dimensionality." This work addresses these challenges by automating the design of efficient spatio-temporal modules through a differentiable search strategy.
Methods
The proposed framework makes NAS feasible for video through three key innovations:
- Differentiable Search Space: Instead of a discrete search, the authors use a continuous relaxation method. The neural module is modeled as a Directed Acyclic Graph (DAG) where connections are weighted and optimized via gradient descent, allowing the best operators to be "strengthened" while others fade out.
- Pseudo-3D (P3D) Operators: To reduce parameter count, 3D convolutions are decoupled into 2D spatial filters followed by 1D temporal filters. This maintains spatio-temporal learning capabilities while significantly lowering computational costs.
- Temporal Segment Sampling: Following the Temporal Segment Network (TSN) strategy, the model samples frames from divided segments to capture long-range dependencies without the memory overhead of processing every single frame.
Results and Discussion
The search process was highly efficient, completing in just 25 hours on a single Nvidia V100 GPU. The discovered architecture demonstrates a strong preference for operators with larger perspective fields, such as dilated convolutions, highlighting the importance of spatial context in action recognition.
Quantitatively, the searched model significantly outperforms much larger manually designed architectures when trained from scratch on the UCF101 dataset. As shown in the comparison table below, the model achieves a superior accuracy of 58.6% while maintaining a remarkably small footprint of 7.32 MB, making it highly suitable for edge device deployment.
Conclusion
This research demonstrates that NAS can discover highly efficient and accurate architectures for video tasks, moving away from the "black art" of manual design. The resulting model's extreme sparsity (0.67M parameters) suggests that traditional 3D models are often over-parameterized. Future work includes evaluating the architecture on larger datasets like Kinetics-400 and exploring the benefits of large-scale pre-training for these NAS-discovered "tiny" models.
Summary updated