Temporal Transformer Networks With Self-Supervision for Action Recognition

Yongkang Zhang, Jun Li, Na Jiang, Guoming Wu, Han Zhang, Zhi-Ping Shi, Zhaoxun Liu, Zizhang Wu, Xianglong LiuIEEE Internet of Things Journal 2023 · DOI · OpenReview

TL;DR

This paper introduces Temporal Transformer Networks with Self-supervision (TTSN), a novel approach to video action recognition that tackles the limitations of existing methods in modeling long-range temporal dependencies and reverse motion. By combining an efficient temporal transformer module with a self-supervised learning strategy that leverages reversed video sequences, TTSN achieves state-of-the-art performance on benchmark datasets while maintaining computational efficiency.

What Problem Does This Paper Address?

Video action recognition is crucial for intelligent systems, especially in resource-constrained environments like IoT devices. However, current 2D Convolutional Neural Network (CNN) based methods struggle with two key challenges: effectively modeling long-range, non-linear temporal relationships between video frames and understanding reverse motion, which can be vital for distinguishing similar-looking actions (e.g., "sit" vs. "stand"). Existing approaches often rely on computationally expensive optical flow or 3D CNNs, or they oversimplify temporal modeling, leading to performance degradation, particularly on complex or subtly different actions.

Methods

The proposed Temporal Transformer Network with Self-supervision (TTSN) addresses these issues through two main components:

  • Efficient Temporal Transformer Module (ETT): This module is designed to capture complex, non-linear temporal dependencies across non-local frames. It incorporates a positional encoding sub-module and a temporal transformer encoder sub-module. Unlike standard transformers, it's optimized for efficiency, focusing on extracting motion-sensitive pixel information.
  • Temporal Sequence Self-supervision Module (TSS): This module uses unlabeled data to learn robust representations of reverse motion. It employs a "random batch random channel" strategy to reverse video sequences, forcing the network to learn to distinguish actions based on their temporal order. This self-supervised approach enhances the model's generalization capabilities and its ability to differentiate actions with similar spatial appearances but reversed temporal dynamics.

The TTSN architecture integrates these modules with a ResNet-based backbone. During training, both ETT and TSS are utilized, while during inference, only the ETT module is employed for efficiency.

Datasets

The paper evaluates TTSN on three widely used action recognition datasets:

  • HMDB51: Contains 51 action categories and 6849 videos.
  • UCF101: Features 101 action categories and 13320 videos.
  • Something-Something V1: A larger dataset with 174 action categories and 108499 videos, known for its emphasis on temporal relationships.

Key Insights and Ideas

  • Reverse Motion is Crucial: The paper highlights that actions which are spatially similar but temporally reversed (e.g., "sit" vs. "stand") are a significant challenge for existing models. The TSS module's ability to learn from reversed sequences is key to overcoming this.
  • Self-Supervision for Robustness: Leveraging unlabeled data through the TSS module significantly improves the model's generalization and its ability to handle complex actions, demonstrating the power of self-supervised learning in this domain.
  • Transformer Efficiency for Video: The "Efficient Temporal Transformer Module" shows that transformer architectures can be adapted for video analysis without prohibitive computational costs, especially when optimized for motion-sensitive regions.
  • Randomness Enhances Performance: The study found that the "Random batch random channel-Reverse" (RR) algorithm for the TSS module, which offers the highest randomness, leads to the best performance and reduced GPU usage.

Key Results

  • State-of-the-Art Performance: TTSN achieves state-of-the-art accuracy on HMDB51 and UCF101 datasets, outperforming previous 2D CNN-based methods and even some 3D CNN models.
  • On HMDB51, TTSN with ImageNet pre-training achieves 60.2% Top-1 accuracy, and with ImageNet+Kinetics pre-training, it reaches 80.2%.
  • On UCF101, TTSN with ImageNet pre-training achieves 86.4% Top-1 accuracy, and with ImageNet+Kinetics pre-training, it reaches 96.8%.
  • Strong Performance on Complex Datasets: On the challenging Something-Something V1 dataset, TTSN achieves 52.4% Top-1 accuracy with 8 frames and 53.3% with 16 frames, demonstrating its powerful temporal modeling capabilities.
  • Computational Efficiency: The ETT and TSS modules add minimal computational overhead. The paper reports GFLOPs of 6.58 for ETT and 1.62 for TSS (with 16 frames), and parameter counts of 2.20M for ETT and 12.59M for TSS. This makes TTSN suitable for resource-constrained environments.
  • Improved Accuracy on Confusing Actions: Visualizations and category-specific accuracy analyses show significant improvements in distinguishing actions like "sit" vs. "stand," and complex actions like "sword," "dribble," and "surfing."

Limitations and Future Work

  • Hidden Dimension Tuning: The paper notes that the optimal hidden dimension for the temporal transformer encoder sub-module (l) is crucial for performance. While they found l=12544 to be optimal, this might require careful tuning for different architectures or datasets.
  • Inference Time: While efficient, TTSN's inference time is slightly slower than some previous models, though still considered real-time.
  • Future Work: The authors suggest exploring other self-supervision algorithms and further optimizing the ETT module. The paper also implies that the computational efficiency and high performance of TTSN make it a strong candidate for deployment in IoT scenarios.

Summary updated

Abstract

In recent years, Internet of Things (IoT) has made rapid development, and IoT devices are developing toward intelligence. IoT terminal devices represented by surveillance cameras play an irreplaceable role in modern society, most of them are integrated with video action recognition and other intelligent functions. However, their performance is somewhat affected by the limitation of computing resources of IoT terminal devices and the lack of long-range nonlinear temporal relation modeling and reverse motion information modeling. To address this urgent problem, we introduce a startling temporal transformer network with self-supervision (TTSN). Our high-performance TTSN mainly consists of a temporal transformer module and a temporal sequence self-supervision (TSS) module. Concisely speaking, we utilize the efficient temporal transformer module to model the nonlinear temporal dependencies among nonlocal frames, which significantly enhances complex motion feature representations. The TSS module we employ unprecedentedly adopts the streamlined strategy of “random batch random channel” to reverse the sequence of video frames, allowing robust extractions of motion information representation from inversed temporal dimensions and improving the generalization capability of the model. Extensive experiments on three widely used data sets (HMDB51, UCF101, and Something–Something V1) have conclusively demonstrated that our proposed TTSN is promising as it successfully achieves state-of-the-art performance for video action recognition. Our TTSN provides the possibility for its application in IoT scenarios due to its computational complexity and high performance.
BibTeX
@article{zhang2023temporal,
  author = {Yongkang Zhang and Jun Li and Na Jiang and Guoming Wu and Han Zhang and Zhi-Ping Shi and Zhaoxun Liu and Zizhang Wu and Xianglong Liu},
  title = {{Temporal Transformer Networks With Self-Supervision for Action Recognition}},
  journal = {IEEE Internet of Things Journal},
  year = {2023},
  doi = {10.1109/jiot.2023.3257992},
}
Figure