Temporal Cycle-Consistency Learning

Debidatta Dwibedi, Yusuf Aytar, Jonathan Tompson, Pierre Sermanet, Andrew Zisserman

Abstract

We introduce a self-supervised representation learning method based on the task of temporal alignment between videos. The method trains a network using temporal cycle consistency (TCC), a differentiable cycle-consistency loss that can be used to find correspondences across time in multiple videos. The resulting per-frame embeddings can be used to align videos by simply matching frames using the nearest-neighbors in the learned embedding space. We show that the learned embeddings enable few-shot classification of action phases and are complementary to other self-supervised methods. The embeddings are also used for applications including transfer of synchronized modalities, synchronized playback, and anomaly detection.

Introduction: The Challenge of Fine-Grained Action Understanding

While deep learning has become excellent at identifying what is happening in a video (e.g., "this is a person pouring water"), it often struggles with the fine-grained stages of an action (e.g., "the container is exactly 50% tilted"). To teach machines these nuances, researchers typically manually label every single frame—a process that is tedious, expensive, and often ambiguous.

Temporal Cycle-Consistency (TCC) addresses this by learning temporal "phases" automatically. By observing how different instances of the same action relate to one another, TCC finds the common semantic thread linking variations in speed, appearance, and camera angle without requiring dense human supervision.

Methods: Learning the Flow of Time

The core innovation is Cycle-Consistency. The model is trained to find matching moments across different videos and "cycle back" to the original starting point.

Cycle-consistency mechanism: finding matches across videos and cycling back to the origin to minimize error.

The process works as follows:

  1. Find a Match: A frame in Video A (the "query") is matched to its most similar counterpart in Video B using a learned embedding space.
  2. Cycle Back: From that match in Video B, the model finds the most similar match back in Video A.
  3. The Loss: If the model is accurate, the "cycled-back" point should land on the original starting frame. Any deviation creates a "cycle-consistency error" used for training.

The authors introduce two differentiable versions of this logic: Cycle-back Classification (CBC), which treats frames as discrete categories, and Cycle-back Regression (CBR). CBR uses a Gaussian prior to penalize the model based on temporal distance, teaching the network that being one frame off is significantly better than being ten frames off.

Datasets and Evaluation

TCC was evaluated on two primary datasets with new, dense annotations:

  • Pouring: Focused on object interaction (reaching, tilting, liquid exiting).
  • Penn Action: Focused on human sports (baseball swings, bench presses, squats).

The authors added "key event" and "phase" labels to measure how well the model understands the progression of an action.

Key Results: Efficiency and Performance

TCC demonstrates remarkable efficiency in learning from limited data. A model trained with only one labeled video can match the performance of a standard supervised model trained with 50 labeled videos.

Classification accuracy vs. number of labeled videos, demonstrating TCC's superior few-shot performance across different sports.

Key findings include:

  • Superior Alignment: TCC outperformed existing self-supervised methods like Shuffle and Learn (SAL) and Time-Contrastive Networks (TCN) across phase classification tasks.
  • Anomaly Detection: By measuring deviations from a standard cycle-consistent path, the model can detect failures in actions, such as a missed rep in a bench press.
  • Cross-Modal Transfer: The learned embeddings allow for "hallucinating" synchronized modalities, such as retrieving the sound of water pouring for a silent video by matching it to a visually similar sequence.

Conclusion

Temporal Cycle-Consistency Learning provides a robust framework for self-supervised video representation. By leveraging the inherent temporal structure of actions across different videos, it reduces the need for expensive human labeling while enabling fine-grained understanding. Future work may extend this to periodic motions or multi-agent scenes where cycle-back logic faces greater ambiguity.

Summary updated

Abstract

We introduce a self-supervised representation learning method based on the task of temporal alignment between videos. The method trains a network using temporal cycle consistency (TCC), a differentiable cycle-consistency loss that can be used to find correspondences across time in multiple videos. The resulting per-frame embeddings can be used to align videos by simply matching frames using the nearest-neighbors in the learned embedding space. To evaluate the power of the embeddings, we densely label the Pouring and Penn Action video datasets for action phases. We show that (i) the learned embeddings enable few-shot classification of these action phases, significantly reducing the supervised training requirements; and (ii) TCC is complementary to other methods of self-supervised learning in videos, such as Shuffle and Learn and Time-Contrastive Networks. The embeddings are also used for a number of applications based on alignment (dense temporal correspondence) between video pairs, including transfer of metadata of synchronized modalities between videos (sounds, temporal semantic labels), synchronized playback of multiple videos, and anomaly detection. Project webpage: https://sites.google.com/view/temporal-cycle-consistency .
BibTeX
@inproceedings{dwibedi2019temporal,
  author = {Debidatta Dwibedi and Yusuf Aytar and Jonathan Tompson and Pierre Sermanet and Andrew Zisserman},
  title = {{Temporal Cycle-Consistency Learning}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2019},
  doi = {10.1109/cvpr.2019.00190},
}
Figure