Abstract
Vision Transformers (ViTs) have shown great promise in video understanding, yet their application to fine-grained action recognition remains challenging. This paper investigates training recipes to optimize Video Vision Transformers (ViViT) for the EPIC-KITCHENS-100 dataset. By exploring initialization, data augmentation, and resolution scaling, we achieve a significant performance boost. Our findings reveal a fundamental architectural trade-off: while transformers excel at noun prediction (object recognition), convolutional networks remain superior for verb prediction (motion). We leverage these complementary strengths through an ensemble approach, achieving state-of-the-art results.
Introduction
Action recognition in egocentric video, such as the EPIC-KITCHENS-100 dataset, requires identifying both the action (verb) and the object (noun). While Video ViTs are powerful, they are data-hungry and sensitive to training configurations. This work provides a systematic "recipe" to enhance ViViT performance for fine-grained action recognition, bridging the gap between standard transformer performance and specialized dataset requirements.
Methodology: A Stronger Training Recipe
The core of our approach is a multi-stage training strategy for the ViViT-B/16x2 model. Our recipe focuses on three primary pillars:
- Multi-stage Initialization: We found that hierarchical pre-training is crucial. Models are initialized from ImageNet-21K, then fine-tuned on Kinetics-400 and Kinetics-700 before reaching EPIC-KITCHENS-100. This provides a much stronger spatial and temporal foundation.
- Advanced Augmentation (CJ+): To prevent overfitting, we move beyond simple cropping. We incorporate heavy-duty strategies like Mixup, Cutmix, and Random Erasing.
- Resolution Scaling: Increasing input resolution from the standard 224x224 up to 384x384 provides consistent gains, though returns begin to diminish at the highest levels.
Key Insights and Results
Our optimized ViViT model reaches 47.4% action accuracy on the validation set, a 3.4% improvement over the original baseline. A striking discovery is the "Noun-Verb Split": Transformers are "Noun Specialists." Their global attention mechanisms make them exceptionally good at identifying objects. However, they surprisingly underperform traditional Convolutional Neural Networks (CNNs) on verb prediction (motion).
To capitalize on these differences, we ensembled our ViViT models with improved CNN architectures, specifically ir-CSN-152 and SlowFast networks. We applied similar training enhancements to these CNNs to ensure a strong baseline for the ensemble.
Conclusion
By combining a robust training recipe for Video Transformers with the complementary temporal strengths of CNNs, we achieved 48.5% action accuracy on the official EPIC-KITCHENS-100 test set. While transformers have revolutionized object recognition in video, the "verb gap" suggests that current architectures still lack the inductive bias necessary to capture fine-grained temporal dynamics as effectively as 3D convolutions. Future work will focus on architectural designs that better integrate temporal information directly into the transformer framework.
Summary updated