Exploring Temporal Granularity in Self-Supervised Video Representation Learning

Rui Qian, Yeqing Li, Liangzhe Yuan, Boqing Gong, Ting Liu, Matthew Brown, Serge J. Belongie, Ming-Hsuan Yang, Hartwig Adam, Yin CuiarXiv 2021 · arXiv

Introduction

Self-supervised learning (SSL) in the video domain has traditionally borrowed heavily from image-based instance discrimination. This approach encourages different temporal views of the same video to share nearly identical feature representations, a property known as "temporal persistency." While effective for general action recognition (e.g., identifying "swimming"), this paradigm often fails for tasks requiring high temporal precision, such as event boundary detection.

Figure 1: Comparison of fine-grained event boundary detection versus persistent video-level recognition.

The TeG (Temporal Granularity) framework addresses this fundamental tension by explicitly modeling both fine-grained temporal details and long-term persistence, allowing for a more nuanced understanding of video content.

The TeG Framework

TeG is designed to capture both the "forest" and the "trees" of a video sequence through three core components:

  1. Long-Short Contained Sampling: To maintain a consistent spatiotemporal context, the model samples a "long clip" and a "short clip," where the short clip is entirely contained within the temporal span of the long clip.
  2. Dual Projection Heads: The video encoder outputs are processed by two specialized heads:
  3. Fine-Grained Head ($g_f$): Employs dense contrastive learning to match specific segments of the short clip to their exact temporal counterparts in the long clip.
  4. Persistent Head ($g_p$): Uses global average pooling to generate a single summary vector, focusing on the overarching theme or action of the video.
  5. Balanced Loss Function: The total loss is a weighted sum of the fine-grained ($\mathcal{L}_f$) and persistent ($\mathcal{L}_p$) objectives, controlled by a hyperparameter $\alpha \in [0, 1]$.
Figure 2: Overview of the TeG framework featuring Long-Short Contained Sampling and dual projection heads.

Experimental Setup and Datasets

The authors evaluated TeG across a diverse set of eight benchmarks to test its versatility:

  • VidSitu: Movie-based event classification.
  • Kinetics-GEBD: Generic Event Boundary Detection.
  • Kinetics-400: Standard action recognition.
  • Downstream Tasks: Something-Something v2 (SSv2), Diving48, UCF101, HMDB51, and AVA-Kinetics for spatiotemporal localization.

Key Insights

Task-Specific Granularity

The research demonstrates that temporal granularity is highly task-specific. Fine-grained features are essential for boundary detection but can introduce "noise" in standard action recognition tasks. Conversely, persistent features provide the global context needed for general labels but are insensitive to precise temporal shifts.

Redefining "Temporal Awareness"

Interestingly, tasks like SSv2 and Diving48, which are typically thought to require fine-grained "temporal awareness" (e.g., distinguishing "opening" from "closing"), actually benefited more from persistent features. This suggests that "temporal awareness" in these contexts may rely more on global temporal context than on high-frequency frame-to-frame changes.

Superiority over Supervised Learning

TeG-FG (the fine-grained variant) outperformed supervised pre-training on Kinetics-GEBD by 8.9%. This indicates that manual labels can sometimes hinder a model's ability to learn the intrinsic temporal structure of video data.

Results and Performance

TeG achieved state-of-the-art results on several fronts:

  • Diving48: Set a new record of 83.6% accuracy, surpassing all previous supervised and unsupervised methods.
  • AVA-Kinetics: Achieved 28.7 mAP, an 8.9-point improvement over the supervised R3D-50 baseline.
  • Efficiency: Reached 67.8% on Kinetics linear evaluation without the need for computationally expensive multi-clip sampling strategies.

Conclusion and Future Directions

TeG proves that a "one size fits all" approach to temporal representation is suboptimal. By balancing fine-grained and persistent features, the framework achieves broad success across varied video tasks. Future work aims to develop a "pyramid of representations" that can automatically adapt its temporal granularity to the specific requirements of a given task, removing the need for manual hyperparameter tuning.

Summary updated

Abstract

This work presents a self-supervised learning framework named TeG to explore Temporal Granularity in learning video representations. In TeG, we sample a long clip from a video and a short clip that lies inside the long clip. We then extract their dense temporal embeddings. The training objective consists of two parts: a fine-grained temporal learning objective to maximize the similarity between corresponding temporal embeddings in the short clip and the long clip, and a persistent temporal learning objective to pull together global embeddings of the two clips. Our study reveals the impact of temporal granularity with three major findings. 1) Different video tasks may require features of different temporal granularities. 2) Intriguingly, some tasks that are widely considered to require temporal awareness can actually be well addressed by temporally persistent features. 3) The flexibility of TeG gives rise to state-of-the-art results on 8 video benchmarks, outperforming supervised pre-training in most cases.
BibTeX
@misc{qian2021exploring,
  author = {Rui Qian and Yeqing Li and Liangzhe Yuan and Boqing Gong and Ting Liu and Matthew Brown and Serge J. Belongie and Ming-Hsuan Yang and Hartwig Adam and Yin Cui},
  title = {{Exploring Temporal Granularity in Self-Supervised Video Representation Learning}},
  year = {2021},
  eprint = {2112.04480},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2112.04480},
}
Figure