Cezsar: A Contrastive Embedding Method for Zero-Shot Action Recognition

Valter Estevam, Rayson Laroca, Hélio Pedrini, David MenottiSSRN 2023 · DOI

TL;DR

This paper introduces CEZSAR, a novel contrastive learning method for Zero-Shot Action Recognition (ZSAR). CEZSAR effectively bridges the semantic gap between visual and textual representations by learning a joint embedding space, leading to state-of-the-art performance on challenging benchmarks.

What Problem Does This Paper Address?

Human action recognition in videos is crucial for many applications, but traditional supervised methods require extensive labeled data. This becomes problematic as new actions emerge constantly. Zero-Shot Action Recognition (ZSAR) aims to address this by enabling models to recognize actions they haven't seen during training. However, ZSAR faces two major challenges: the "semantic gap" (discrepancy between visual features and semantic descriptions) and "domain shift" (differences between training and testing data distributions). Existing methods struggle to effectively align visual and textual information, leading to suboptimal performance.

Methods

CEZSAR tackles the ZSAR challenges by employing a contrastive learning approach within a joint embedding space. The core of the method involves two main modules:

  • Visual Embedding Module (VEM): This module processes video clips. It uses a pre-trained CNN (like ResNet-152) to extract frame-level features, followed by a Transformer encoder to capture temporal information. This results in a dense representation of the video.
  • Sentence Embedding Module (SEM): This module takes natural language descriptions of actions and encodes them using Sentence-BERT (SBERT), a powerful pre-trained model for sentence embeddings.

The key innovation lies in training these modules to project video and text representations into a shared "joint semantic space." The training objective is to pull representations of a video and its corresponding positive description closer together, while pushing apart representations of a video and its negative descriptions. To achieve this, the authors propose a hard negative sampling method. This method automatically generates "hard" negative examples (unrelated descriptions for a given video) without requiring human annotation, which is crucial for scaling. This process is trained using a triplet loss function.

Datasets

The authors trained their model using the ActivityNet Captions dataset. This dataset contains untrimmed videos with human-generated temporal segment annotations and descriptions.

For evaluation, they used two widely recognized action recognition datasets:

  • UCF-101: Contains 13,320 videos across 101 action classes.
  • Kinetics-400: A larger dataset with 306,245 videos across 400 action classes.

They evaluated their model under various "split configurations," which define the proportion of seen and unseen classes during training and testing.

Key Insights and Ideas

  • Contrastive learning is effective for ZSAR: The paper demonstrates that contrastive learning is a powerful tool for bridging the semantic gap between modalities in ZSAR, outperforming previous approaches.
  • Textual semantics are more robust to domain shift: The authors hypothesize that textual descriptions are less affected by domain shift than visual features. By conditioning visual learning on text, they effectively mitigate this problem.
  • Automatic hard negative sampling is crucial: Generating effective negative samples is vital for contrastive learning. Their proposed method automates this process, significantly reducing the need for manual annotation and enabling efficient training.
  • Joint embedding space is key: Projecting both visual and textual information into a common space allows for direct comparison and alignment, which is fundamental to solving the semantic gap.
  • Visual features alone can be competitive: Even when using only visual features (VE) without explicit object semantics, their model achieved strong results, highlighting the power of their learned joint embedding space.

Key Results

CEZSAR achieved state-of-the-art results on both the UCF-101 and Kinetics-400 datasets across various split configurations. For instance, on UCF-101 with 101 test classes, their model achieved 72.6% accuracy using visual features alone, and 72.9% when incorporating object-level semantic information. On Kinetics-400 with 400 test classes, they reached 23.8% accuracy with visual features and object semantics, outperforming previous methods. The paper also shows that their model significantly reduces confusion between visually similar actions, as evidenced by confusion matrices.

Limitations and Future Work

The authors acknowledge a few limitations and suggest future research directions:

  • Influence of pre-training dataset size: They plan to investigate how the size of the pre-training dataset impacts contrastive learning performance.
  • Using images as label prototypes: They intend to explore the impact of incorporating images as label prototypes on reducing the semantic gap.
  • Object semantics not always beneficial: In some cases, particularly with highly similar actions, the inclusion of object semantics did not lead to improvements and sometimes even increased confusion. This suggests that the effectiveness of object semantics might depend on the specific action classes.
  • Computational efficiency: While their method is more efficient than some prior works (e.g., CLIP-based models), further optimization for training speed and resource usage could be explored.

Summary updated

BibTeX
@misc{estevam2023cezsar,
  author = {Valter Estevam and Rayson Laroca and Hélio Pedrini and David Menotti},
  title = {{Cezsar: A Contrastive Embedding Method for Zero-Shot Action Recognition}},
  year = {2023},
  doi = {10.2139/ssrn.4333781},
  howpublished = {SSRN},
}
Figure