Abstract
Deep Neural Networks (DNNs) typically require vast amounts of labeled data to achieve high performance in video action recognition. In few-shot scenarios, these models often suffer from overparameterization and an inability to capture complex temporal dynamics. This paper introduces Temporal Attention Vectors (TAVs), a lightweight supervised learning framework that utilizes manually initialized temporal priors and a shallow importance learner. By reducing trainable parameters to approximately 2.1M, the proposed method achieves state-of-the-art results on standard benchmarks with as few as 8–10 training videos per class.
Introduction
Achieving high accuracy in video action recognition usually necessitates thousands of manually labeled examples. Standard deep models, often possessing tens of millions of parameters, tend to "memorize" small training sets rather than learning generalizable temporal patterns. Furthermore, capturing how an action evolves over time—the temporal complexity—remains a significant hurdle when data is scarce. This research proposes a way to encode video-wide temporal correlations without requiring massive datasets or external metadata.
Methodology
The core of the proposed architecture is the Temporal Attention Vector (TAV) framework. The system avoids overfitting by using a "frozen" backbone (such as ResNet-152 or I3D pre-trained on ImageNet/Kinetics) to extract frame features. These features are then aggregated using TAVs.
As illustrated in the architecture above, the process involves three main stages:
- Feature Extraction: Video frames pass through a frozen pre-trained CNN to generate frame features.
- TAV Initialization: Vectors are initialized to act as "temporal spotlights," focusing on different segments and durations of the video.
- Importance Score Learner: A shallow CNN learns to assign weights to these TAVs, identifying which temporal segments are most critical for a specific action.
Temporal Attention Vector Initialization
A key insight of this work is that manual priors can outperform pure learning in data-scarce environments. The authors explored several initialization strategies for TAVs to bake in logical assumptions about how actions occur in continuous temporal chunks.
The most successful strategy, Short-Long Dynamic Gaussian (SLDG), creates vectors that cover multiple temporal scales. This allows the model to capture both split-second movements and actions that span the entire video duration. By using Gaussian shapes, the model assumes actions are localized in time, which prevents the importance learner from having to "learn" basic temporal structures from scratch.
Experimental Results
The TAV approach was evaluated on the UCF101 and HMDB51 benchmarks using limited training samples (1, 2, 4, 8, or 10 videos per class):
- UCF101: With only 10 videos per class, the model achieved 92% accuracy, significantly higher than the previous state-of-the-art of 84.5%.
- HMDB51: With 8 videos per class, it reached 59% accuracy, outperforming the Temporal Structure Filter (TSF) at 50.3%.
- Efficiency: The model utilizes only 2.1M trainable parameters, a 97% reduction compared to standard spatiotemporal networks (e.g., 85.8M parameters), which effectively mitigates overfitting.
Conclusion
By leveraging temporal priors through SLDG-initialized TAVs and maintaining a low parameter count, this method demonstrates that high-performance video action recognition is possible even with extremely limited data. Future work may involve applying TAVs to broader tasks such as video summarization or temporal action localization, and reducing the current dependency on pre-computed optical flow.
Summary updated