Learning Deep Intrinsic Video Representation by Exploring Temporal Coherence and Graph Structure

Yingwei Pan, Yehao Li, Ting Yao, Tao Mei, Houqiang Li, Yong Rui

Introduction

Standard image-based deep learning models often treat video frames as independent entities, ignoring the inherent sequential and topological structure of video data. This paper introduces Temporal and Graph-structured Feature Learning (TGFL), an unsupervised framework designed to learn "intrinsic" video representations. The core challenge addressed is balancing local temporal redundancy—where adjacent frames are similar—with global correlation, where semantically identical scenes may recur at distant points in time.

The TGFL Framework

The TGFL framework integrates two distinct priors into a Deep Convolutional Neural Network (DCNN) to refine feature embeddings without manual labels.

The TGFL framework integrates temporal coherence and graph structure preservation into a deep learning architecture to learn intrinsic video representations.

Modeling Temporal Coherence (Local)

The model employs a triplet ranking loss to ensure local coherence. For any given query frame, the system identifies "positive" and "negative" frames. Unlike random sampling, TGFL uses a rigorous triplet selection mechanism: a frame is only considered positive if it is both temporally adjacent and visually similar. This prevents camera shakes or rapid motion blur from introducing noise into the learning process.

Graph Structure Preservation (Holistic)

To capture the global topology of a video, the authors treat the entire sequence as a graph where every frame is a node. By estimating an affinity matrix and applying a Graph Laplacian regularizer, the network is forced to map visually similar frames to nearby points in the embedding space, even if they are temporally far apart. This allows the model to "remember" and group recurring scenes, such as a specific location in a vlog.

Experimental Results

The model was evaluated on the Columbia Consumer Videos (CCV) and YouTube Highlight datasets, focusing on retrieval, classification, and highlight detection tasks.

Qualitative comparison of video retrieval results. TGFL (bottom row) identifies more semantically relevant frames compared to standard features and partial models.

Key performance metrics include:

  • Video Retrieval: Achieved a 72.13% Mean Average Precision (MAP), a significant 6.8% improvement over standard AlexNet fc6 features.
  • Video Classification: Reached 65.26% MAP, outperforming temporal-only embedding baselines.
  • Highlight Detection: Boosted MAP to 48.08%, a 12.1% relative improvement over the baseline.

Ablation studies demonstrated that while temporal coherence (TC) and graph structure (GS) are individually beneficial, their fusion (TGFL) consistently yields the best results, proving they provide complementary information about the video's intrinsic properties.

Conclusion

TGFL demonstrates that unsupervised learning can effectively capture complex video dynamics by leveraging both local and global structural priors. While the current model relies on frame-level representations and simple pooling for video-level tasks, it sets a strong foundation for future work involving Recurrent Neural Networks (RNNs) to better model long-term temporal dynamics.

Summary updated

Abstract

Learning video representation is not a trivial task, as video is an information-intensive media where each frame does not exist independently. Locally, a video frame is visually and semantically similar with its adjacent frames. Holistically, a video has its inherent structure--the correlations among video frames. For example, even the frames far from each other may also hold similar semantics. Such context information is therefore important to characterize the intrinsic representation of a video frame. In this paper, we present a novel approach to learn the deep video representation by exploring both local and holistic contexts. Specifically, we propose a triplet sampling mechanism to encode the local temporal relationship of adjacent frames based on their deep representations. In addition, we incorporate the graph structure of the video, as a priori, to holistically preserve the inherent correlations among video frames. Our approach is fully unsupervised and trained in an end-to-end deep convolutional neural network architecture. By extensive experiments, we show that our learned representation can significantly boost several video recognition tasks (retrieval, classification, and highlight detection) over traditional video representations.
BibTeX
@inproceedings{pan2016learning,
  author = {Yingwei Pan and Yehao Li and Ting Yao and Tao Mei and Houqiang Li and Yong Rui},
  title = {{Learning Deep Intrinsic Video Representation by Exploring Temporal Coherence and Graph Structure}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2016},
}
Figure