Self-Supervised Video Representation Learning with Constrained Spatiotemporal Jigsaw

Yuqi Huo, Mingyu Ding, Haoyu Lu, Zhiwu Lu, Tao Xiang, Ji-Rong Wen, Ziyuan Huang, Jianwen Jiang, Shiwei Zhang, Mingqian Tang, Songfang Huang, Ping Luo

TL;DR

This paper introduces "Constrained Spatiotemporal Jigsaw" (CSJ), a novel self-supervised learning method for videos. CSJ tackles the challenge of learning video representations by creating solvable 3D jigsaw puzzles that capture spatiotemporal continuity, outperforming prior methods on action recognition and video retrieval tasks.

What Problem Does This Paper Address?

Self-supervised learning (SSL) has seen great success in image analysis, but its application to videos has lagged. Videos are inherently spatiotemporal, meaning both spatial content and temporal progression are crucial. Existing SSL methods for videos often focus on either spatial or temporal aspects independently, failing to capture this joint spatiotemporal understanding. The authors propose that learning to understand spatiotemporal continuity is key for effective video representation. A natural approach, solving 3D spatiotemporal jigsaw puzzles, is computationally intractable due to the vast number of possible permutations. This paper aims to bridge this gap by developing a solvable yet effective SSL pretext task that leverages spatiotemporal continuity.

Methods

The core idea is to create "Constrained Spatiotemporal Jigsaws" (CSJ). Instead of randomly shuffling video clips into 3D jigsaw pieces, CSJ employs a constrained shuffling strategy. This strategy ensures that the shuffled clips still contain large, continuous spatiotemporal cuboids (LCCs), which act as strong cues for understanding continuity.

Since directly solving these constrained jigsaws is still difficult, the authors propose four surrogate tasks that are more tractable but still encourage learning spatiotemporal continuity:

  1. Largest Continuous Cuboid Detection (LCCD): The model learns to identify the locations of the top-2 largest continuous cuboids within a shuffled clip.
  2. Clip Shuffling Pattern Classification (CSPC): The model classifies which of the 8 possible constrained shuffling patterns was applied to a given clip.
  3. Contrastive Learning over Shuffled Clips (CLSC): A contrastive loss is used to distinguish between the original clip and the most "continuous" shuffled clip (positive pair) versus other shuffled clips (negative pairs).
  4. Clip Continuity Measure Regression (CCMR): The model regresses a measure of global continuity for the shuffled clip, based on factors like the volume of LCCs and the degree of shuffling.

These four tasks are trained jointly with adaptive weighting and curriculum learning.

Datasets

The paper uses standard video datasets for pre-training and evaluation:

  • UCF101: Used for pre-training and evaluation.
  • HMDB51: Used for pre-training and evaluation.
  • Kinetics-400 (K400): Used for pre-training.
  • Breakfast dataset: Used for qualitative evaluation on temporal action segmentation.

Key Insights and Ideas

  • Spatiotemporal Continuity is Key: The paper strongly advocates that learning to understand how spatial and temporal information cohere is fundamental for video SSL, moving beyond purely spatial or temporal pretext tasks.
  • Intractability of 3D Jigsaws: Directly solving 3D spatiotemporal jigsaws is computationally infeasible, motivating the need for alternative approaches.
  • Constrained Jigsaws as a Bridge: The "constrained" aspect is crucial. By ensuring large LCCs remain, the problem becomes more manageable while retaining essential continuity information.
  • Surrogate Tasks for Solvability: The clever design of four surrogate tasks allows the model to learn about continuity without the combinatorial explosion of full jigsaw solving.
  • Local vs. Global Continuity: The tasks are designed to capture both local continuity (LCCD, CSPC) and global continuity (CLSC, CCMR), providing a comprehensive understanding.

Key Results

  • State-of-the-Art Performance: CSJ achieves state-of-the-art results on action recognition (UCF101, HMDB51) and video retrieval (UCF101) tasks.
  • Competitive with Strong Baselines: On UCF101, CSJ achieves 70.4% accuracy with a weaker backbone (R2D3D-18) compared to other methods. It also outperforms methods using only temporal or spatial transformations, and even those using contrastive learning without explicit spatiotemporal transformations.
  • Close to Upper Bound: CSJ's performance is noted to be close to fully-supervised learning, validating its effectiveness.
  • Effective Surrogate Tasks: Ablation studies show that each surrogate task contributes to performance, and their joint training is most effective. Adaptive weighting and curriculum learning further boost results.
  • Qualitative Improvements: Visualizations show that CSJ-pretrained models better capture spatiotemporal information, leading to improved performance in tasks like temporal action segmentation.

Limitations and Future Work

  • Direct Jigsaw Solving: The paper acknowledges that directly solving the CSJ task is less effective than using the surrogate tasks, suggesting that the surrogate tasks are a more practical approach for learning useful representations.
  • Computational Cost: While more tractable than unconstrained jigsaws, the method still involves significant computation for pre-training.
  • Future Work: The paper implicitly suggests that further exploration of constrained jigsaw designs and surrogate tasks could lead to even better video representations. The success on temporal action segmentation also points to its potential for other complex video understanding tasks.

Summary updated

Abstract

This paper proposes a novel pretext task for self-supervised video representation learning by exploiting spatiotemporal continuity in videos. It is motivated by the fact that videos are spatiotemporal by nature and a representation learned to detect spatiotemporal continuity/discontinuity is thus beneficial for downstream video content analysis tasks. A natural choice of such a pretext task is to construct spatiotemporal (3D) jigsaw puzzles and learn to solve them. However, this task turns out to be intractable. We thus propose Constrained Spatiotemporal Jigsaw (CSJ) whereby the 3D jigsaws are formed in a constrained manner to ensure that large continuous spatiotemporal cuboids exist in a shuffled clip to provide sufficient cues for the model to reason about the continuity. With the constrained jigsaw puzzles, instead of solving them directly, which could still be extremely hard, we carefully design four surrogate tasks that are more solvable but meanwhile still ensure that the learned representation is sensitive to spatiotemporal continuity at both the local and global levels. Extensive experiments show that our CSJ achieves state-of-the-art on two downstream tasks across various benchmarks.
BibTeX
@inproceedings{huo2021self,
  author = {Yuqi Huo and Mingyu Ding and Haoyu Lu and Zhiwu Lu and Tao Xiang and Ji-Rong Wen and Ziyuan Huang and Jianwen Jiang and Shiwei Zhang and Mingqian Tang and Songfang Huang and Ping Luo},
  title = {{Self-Supervised Video Representation Learning with Constrained Spatiotemporal Jigsaw}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.24963/ijcai.2021/104},
}
Figure