CycleCL: Self-Supervised Learning for Periodic Videos

Matteo Destro, Michael GygliWACV 2024

TL;DR

This paper introduces CycleCL, a novel self-supervised learning method specifically designed for periodic video data. By cleverly sampling triplets of frames and optimizing a contrastive loss, CycleCL learns representations that are sensitive to the phase of a cycle while being invariant to specific repetitions, outperforming existing methods on various downstream tasks.

What Problem Does This Paper Address?

Analyzing periodic video sequences, such as those found in industrial automation, medical monitoring (e.g., vital signs), or physical training, presents a unique challenge. Standard self-supervised learning methods, often trained on diverse datasets like ImageNet or Kinetics, struggle with periodic data. These methods typically rely on random augmentations to create positive and negative sample pairs, which is suboptimal when dealing with videos where nearly identical frames might repeat within a cycle. This can lead to models that are not sensitive to the subtle changes within a cycle or are easily confused by unrelated noise. The scarcity of labeled data for these specialized applications further necessitates effective self-supervised approaches.

Methods

CycleCL tackles this problem by proposing a novel self-supervised learning method based on a triplet loss. The core idea is to learn representations that are sensitive to the phase of a cycle but invariant to the exact repetition.

The method involves two key iterative steps:

  1. Triplet Sampling: For each "anchor" frame, the system identifies:
  2. Positive frames: Frames from different repetitions that depict the same phase as the anchor.
  3. Negative frames: Frames from different phases within the same or other repetitions. This sampling is guided by a temporal self-similarity matrix (TSM) computed from frame embeddings. Two sampling strategies are proposed: "Min/Max Top-k" and "Mean Thresholding," with the latter showing superior performance.

  4. Encoder Optimization: A triplet loss is minimized to pull positive frames closer to the anchor and push negative frames further away. This process is iterated, allowing the encoder's features to evolve and enable more accurate triplet sampling over time.

To enhance robustness, CycleCL also incorporates an augmentation strategy where positive frames are augmented with transformations (e.g., color jittering, blurring) during the loss computation, forcing the model to learn phase-specific features while ignoring these artificial variations.

Datasets

The paper evaluates CycleCL on four datasets:

  • Industrial dataset: A newly collected dataset of videos from glass packaging production lines, featuring anomalous events.
  • Countix: A subset of the Kinetics dataset, commonly used for repetition counting.
  • QUVA: A smaller dataset of periodic videos.
  • PERTUBE: Another smaller dataset of periodic videos.

These datasets are labeled as either periodic or non-periodic, with the Industrial dataset also containing annotations for anomalies.

Key Insights and Ideas

  • Periodic data requires specialized SSL: Standard self-supervised methods fail on periodic data because their random sampling strategies are ill-suited for identifying subtle phase-based variations amidst repetitive patterns.
  • Phase-sensitivity and repetition-invariance are key: The ideal representation for periodic data should capture the progression within a cycle but treat identical phases across different cycles as equivalent.
  • Iterative refinement of sampling and learning: The core innovation lies in the iterative process where improved feature representations lead to better triplet sampling, which in turn further refines the representations. This creates a virtuous cycle of learning.
  • Smart triplet sampling is crucial: Simply picking adjacent frames or relying on fixed numbers of positives/negatives is insufficient. Dynamic sampling based on similarity, like the "Mean Thresholding" approach, is vital for effective learning.
  • Augmenting positives only is beneficial: Applying augmentations selectively to positive samples makes the learning task harder, forcing the model to become invariant to these transformations and thus more robust.

Key Results

  • CycleCL significantly outperforms state-of-the-art self-supervised learning methods (SimCLR, DINO, TCC, RepNet) on all four evaluated datasets for both nearest-neighbor classification and unsupervised anomaly detection tasks.
  • On the Industrial dataset, CycleCL reduces the performance gap to a fully supervised method to 8%, a substantial improvement over the 18% gap achieved by the second-best method.
  • The "Mean Thresholding" triplet sampling strategy, with a specific β value, proved to be the most effective, outperforming other strategies by a significant margin.
  • Augmenting only the positive samples improved the F1-score by 7.5% compared to not using augmentations.
  • For anomaly detection, CycleCL's "cycle" representation (using similarity to other frames) significantly improved performance over raw frame embeddings, achieving an F1-score of 52.9% compared to 27.9% for raw features.

Limitations and Future Work

  • Batch Normalization limitations: The paper notes that performance can be affected by batch size, particularly with very long sequences, suggesting that Batch Normalization might not work well in such scenarios.
  • Supervised gap persists: While CycleCL significantly closes the gap, a performance difference still exists between its unsupervised approach and fully supervised methods on the Industrial dataset.
  • Future work could explore: The authors suggest that further research could investigate more sophisticated augmentation strategies and explore the impact of different sequence lengths and batch sizes on performance. The effectiveness of CycleCL on dynamic and static camera scenes, and across various visual differences, suggests its broad applicability.

Summary updated

BibTeX
@inproceedings{destro2024cyclecl,
  author = {Matteo Destro and Michael Gygli},
  title = {{CycleCL: Self-Supervised Learning for Periodic Videos}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
}
Figure