Towards Training Stronger Video Vision Transformers for EPIC-KITCHENS-100 Action Recognition

Ziyuan Huang, Zhiwu Qing, Xiang Wang, Yutong Feng, Shiwei Zhang, Jianwen Jiang, Zhurong Xia, Mingqian Tang, Nong Sang, Marcelo H. Ang Jr.arXiv 2021 · OpenReview

Abstract

Vision Transformers (ViTs) have shown great promise in video understanding, yet their application to fine-grained action recognition remains challenging. This paper investigates training recipes to optimize Video Vision Transformers (ViViT) for the EPIC-KITCHENS-100 dataset. By exploring initialization, data augmentation, and resolution scaling, we achieve a significant performance boost. Our findings reveal a fundamental architectural trade-off: while transformers excel at noun prediction (object recognition), convolutional networks remain superior for verb prediction (motion). We leverage these complementary strengths through an ensemble approach, achieving state-of-the-art results.

Introduction

Action recognition in egocentric video, such as the EPIC-KITCHENS-100 dataset, requires identifying both the action (verb) and the object (noun). While Video ViTs are powerful, they are data-hungry and sensitive to training configurations. This work provides a systematic "recipe" to enhance ViViT performance for fine-grained action recognition, bridging the gap between standard transformer performance and specialized dataset requirements.

Methodology: A Stronger Training Recipe

The core of our approach is a multi-stage training strategy for the ViViT-B/16x2 model. Our recipe focuses on three primary pillars:

  1. Multi-stage Initialization: We found that hierarchical pre-training is crucial. Models are initialized from ImageNet-21K, then fine-tuned on Kinetics-400 and Kinetics-700 before reaching EPIC-KITCHENS-100. This provides a much stronger spatial and temporal foundation.
  2. Advanced Augmentation (CJ+): To prevent overfitting, we move beyond simple cropping. We incorporate heavy-duty strategies like Mixup, Cutmix, and Random Erasing.
  3. Resolution Scaling: Increasing input resolution from the standard 224x224 up to 384x384 provides consistent gains, though returns begin to diminish at the highest levels.
Ablation study of the ViViT training recipe, showing the impact of initialization, augmentation (CJ+), and resolution on Action (A), Verb (V), and Noun (N) accuracy.

Key Insights and Results

Our optimized ViViT model reaches 47.4% action accuracy on the validation set, a 3.4% improvement over the original baseline. A striking discovery is the "Noun-Verb Split": Transformers are "Noun Specialists." Their global attention mechanisms make them exceptionally good at identifying objects. However, they surprisingly underperform traditional Convolutional Neural Networks (CNNs) on verb prediction (motion).

To capitalize on these differences, we ensembled our ViViT models with improved CNN architectures, specifically ir-CSN-152 and SlowFast networks. We applied similar training enhancements to these CNNs to ensure a strong baseline for the ensemble.

Performance improvements for convolutional models (ir-CSN-152 and SlowFast) using similar training enhancements like resolution scaling and advanced augmentation.

Conclusion

By combining a robust training recipe for Video Transformers with the complementary temporal strengths of CNNs, we achieved 48.5% action accuracy on the official EPIC-KITCHENS-100 test set. While transformers have revolutionized object recognition in video, the "verb gap" suggests that current architectures still lack the inductive bias necessary to capture fine-grained temporal dynamics as effectively as 3D convolutions. Future work will focus on architectural designs that better integrate temporal information directly into the transformer framework.

Summary updated

Abstract

With the recent surge in the research of vision transformers, they have demonstrated remarkable potential for various challenging computer vision applications, such as image recognition, point cloud classification as well as video understanding. In this paper, we present empirical results for training a stronger video vision transformer on the EPIC-KITCHENS-100 Action Recognition dataset. Specifically, we explore training techniques for video vision transformers, such as augmentations, resolutions as well as initialization, etc. With our training recipe, a single ViViT model achieves the performance of 47.4\% on the validation set of EPIC-KITCHENS-100 dataset, outperforming what is reported in the original paper by 3.4%. We found that video transformers are especially good at predicting the noun in the verb-noun action prediction task. This makes the overall action prediction accuracy of video transformers notably higher than convolutional ones. Surprisingly, even the best video transformers underperform the convolutional networks on the verb prediction. Therefore, we combine the video vision transformers and some of the convolutional video networks and present our solution to the EPIC-KITCHENS-100 Action Recognition competition.
BibTeX
@misc{huang2021towards,
  author = {Ziyuan Huang and Zhiwu Qing and Xiang Wang and Yutong Feng and Shiwei Zhang and Jianwen Jiang and Zhurong Xia and Mingqian Tang and Nong Sang and Marcelo H. Ang Jr.},
  title = {{Towards Training Stronger Video Vision Transformers for EPIC-KITCHENS-100 Action Recognition}},
  year = {2021},
  eprint = {2106.05058},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2106.05058},
}
Figure