Temporal Contrastive Pretraining for Video Action Recognition

Guillaume Lorre, Jaonary Rabarisoa, Astrid Orcesi, Samia Ainouz, Stephane CanuWACV 2020

TL;DR

This paper introduces a self-supervised method for learning video representations by predicting future video segments using Contrastive Predictive Coding (CPC). This approach significantly improves action recognition performance, especially when labeled data is scarce, by learning robust temporal features without human annotation.

What Problem Does This Paper Address?

Action recognition, a crucial task in video analysis, typically relies on supervised learning with deep convolutional neural networks. However, acquiring large, labeled video datasets is expensive and time-consuming. This paper addresses the challenge of learning effective video representations from unlabeled data, aiming to reduce the reliance on extensive manual annotation and improve performance in low-data regimes.

Methods

The core of this work is adapting Contrastive Predictive Coding (CPC) for video data. CPC is a self-supervised learning framework that learns representations by predicting future data points from past context. In this paper, the authors propose a task where the model predicts future high-level frame representations given past video segments.

Here's a breakdown of their approach:

  • Input Modalities: They experiment with optical flows, image differences, and raw images as input. Optical flows and image differences are favored as they focus on motion and reduce the influence of static background information.
  • Encoder: A ResNet18 is used as the encoder to extract feature maps from video segments.
  • Autoregressive Model: Temporally masked 3D convolutions form the autoregressive model, which aggregates past feature maps to create a context vector. This context vector summarizes information from previous segments.
  • Prediction: A prediction network uses the context vector to estimate future frame representations.
  • Loss Function: Instead of a direct regression loss, they employ Noise Contrastive Estimation (NCE) loss. NCE frames the problem as a classification task: distinguishing the true future representation from a set of negative examples. This loss is known to maximize mutual information between the context and the predicted future.
  • Negative Examples: To make the task challenging and learn robust representations, they use both "difficult" negative examples (from the same video but different segments) and negative examples from other videos in the batch.
  • Evaluation: The learned representations are evaluated using two methods:
  • Linear Classification: The pre-trained encoder is fixed, and only a linear classifier is trained on top. This measures the inherent quality of the learned representations.
  • Fine-tuning: The entire network (encoder and classifier) is fine-tuned on the downstream task.

Datasets

The authors utilize the following datasets:

  • UCF-101: A widely used dataset for action recognition, containing 101 action categories. Used for both pre-training and evaluation.
  • HMDB51: Another action recognition dataset with 51 action categories, known for being more challenging than UCF-101 due to shorter and lower-quality videos. Used for evaluation.
  • Kinetics: A large-scale dataset with 600 action categories. Used exclusively for pre-training to explore the benefits of larger unlabeled datasets.

Key Insights and Ideas

  • CPC for Video is Effective: The paper demonstrates that CPC, originally successful for audio, text, and images, can be effectively adapted for video and yields strong temporal representations.
  • Predicting Future Representations is Key: By predicting future representations rather than raw pixels, the model is encouraged to learn high-level semantic information about motion and actions, avoiding the pitfalls of low-level generative modeling.
  • The Importance of "Difficult" Negatives: Using negative examples from the same video (but different segments) significantly improves performance, highlighting the need for challenging contrastive tasks.
  • Transferability to Scarce Data: The pre-trained representations are particularly beneficial when labeled data is limited, showing a substantial performance gap over training from scratch.
  • Modality Agnostic: The method shows good performance across different input modalities (optical flows, image differences, images), indicating its general applicability.

Key Results

  • Improved Linear Classification: The proposed method achieves significantly higher accuracy in linear classification on UCF-101 with optical flows (78.4%) compared to random initialization (22.4%) and state-of-the-art methods like AOT [32] (58.6%).
  • Competitive Fine-tuning: Fine-tuning the pre-trained model also yields competitive results, outperforming state-of-the-art on HMDB51 and being comparable on UCF-101.
  • Low-Data Performance: With only 5% of labeled data, the method achieves 80% of the performance obtained with full data, demonstrating its effectiveness in low-resource scenarios.
  • Modality Performance: Optical flows generally yield the best results, but image differences and raw images also see significant improvements with the proposed pre-training compared to training from scratch.
  • Kinetics Pre-training Advantage: Pre-training on the larger Kinetics dataset leads to better performance than pre-training on UCF-101, suggesting that more unlabeled data leads to more robust representations.

Limitations and Future Work

  • Reliance on Pre-calculated Optical Flows: The authors note that the performance is influenced by the quality of the pre-calculated optical flows. They suggest that integrating motion representation learning directly within the network, rather than relying on external pre-processing, could be a promising future direction.
  • Autoregressive Model Choice: While they found temporally masked 3D convolutions to be effective, they explored other autoregressive models like ConvLSTM and Conv(2+1)D, indicating that the choice of the autoregressive component can impact results.
  • Computational Cost: While not explicitly detailed as a limitation, the use of 3D convolutions and large datasets implies significant computational resources are required for pre-training.

The paper concludes by highlighting the generalizability of their method across different inputs and its strong performance in low-data scenarios, paving the way for more efficient video understanding.

Summary updated

BibTeX
@inproceedings{lorre2020temporal,
  author = {Guillaume LORRE and Jaonary Rabarisoa and Astrid Orcesi and Samia Ainouz and Stephane Canu},
  title = {{Temporal Contrastive Pretraining for Video Action Recognition}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2020},
}
Figure