Introduction
Standard image-based deep learning models often treat video frames as independent entities, ignoring the inherent sequential and topological structure of video data. This paper introduces Temporal and Graph-structured Feature Learning (TGFL), an unsupervised framework designed to learn "intrinsic" video representations. The core challenge addressed is balancing local temporal redundancy—where adjacent frames are similar—with global correlation, where semantically identical scenes may recur at distant points in time.
The TGFL Framework
The TGFL framework integrates two distinct priors into a Deep Convolutional Neural Network (DCNN) to refine feature embeddings without manual labels.
Modeling Temporal Coherence (Local)
The model employs a triplet ranking loss to ensure local coherence. For any given query frame, the system identifies "positive" and "negative" frames. Unlike random sampling, TGFL uses a rigorous triplet selection mechanism: a frame is only considered positive if it is both temporally adjacent and visually similar. This prevents camera shakes or rapid motion blur from introducing noise into the learning process.
Graph Structure Preservation (Holistic)
To capture the global topology of a video, the authors treat the entire sequence as a graph where every frame is a node. By estimating an affinity matrix and applying a Graph Laplacian regularizer, the network is forced to map visually similar frames to nearby points in the embedding space, even if they are temporally far apart. This allows the model to "remember" and group recurring scenes, such as a specific location in a vlog.
Experimental Results
The model was evaluated on the Columbia Consumer Videos (CCV) and YouTube Highlight datasets, focusing on retrieval, classification, and highlight detection tasks.
Key performance metrics include:
- Video Retrieval: Achieved a 72.13% Mean Average Precision (MAP), a significant 6.8% improvement over standard AlexNet fc6 features.
- Video Classification: Reached 65.26% MAP, outperforming temporal-only embedding baselines.
- Highlight Detection: Boosted MAP to 48.08%, a 12.1% relative improvement over the baseline.
Ablation studies demonstrated that while temporal coherence (TC) and graph structure (GS) are individually beneficial, their fusion (TGFL) consistently yields the best results, proving they provide complementary information about the video's intrinsic properties.
Conclusion
TGFL demonstrates that unsupervised learning can effectively capture complex video dynamics by leveraging both local and global structural priors. While the current model relies on frame-level representations and simple pooling for video-level tasks, it sets a strong foundation for future work involving Recurrent Neural Networks (RNNs) to better model long-term temporal dynamics.
Summary updated