Introduction
Self-supervised learning (SSL) in the video domain has traditionally borrowed heavily from image-based instance discrimination. This approach encourages different temporal views of the same video to share nearly identical feature representations, a property known as "temporal persistency." While effective for general action recognition (e.g., identifying "swimming"), this paradigm often fails for tasks requiring high temporal precision, such as event boundary detection.
The TeG (Temporal Granularity) framework addresses this fundamental tension by explicitly modeling both fine-grained temporal details and long-term persistence, allowing for a more nuanced understanding of video content.
The TeG Framework
TeG is designed to capture both the "forest" and the "trees" of a video sequence through three core components:
- Long-Short Contained Sampling: To maintain a consistent spatiotemporal context, the model samples a "long clip" and a "short clip," where the short clip is entirely contained within the temporal span of the long clip.
- Dual Projection Heads: The video encoder outputs are processed by two specialized heads:
- Fine-Grained Head ($g_f$): Employs dense contrastive learning to match specific segments of the short clip to their exact temporal counterparts in the long clip.
- Persistent Head ($g_p$): Uses global average pooling to generate a single summary vector, focusing on the overarching theme or action of the video.
- Balanced Loss Function: The total loss is a weighted sum of the fine-grained ($\mathcal{L}_f$) and persistent ($\mathcal{L}_p$) objectives, controlled by a hyperparameter $\alpha \in [0, 1]$.
Experimental Setup and Datasets
The authors evaluated TeG across a diverse set of eight benchmarks to test its versatility:
- VidSitu: Movie-based event classification.
- Kinetics-GEBD: Generic Event Boundary Detection.
- Kinetics-400: Standard action recognition.
- Downstream Tasks: Something-Something v2 (SSv2), Diving48, UCF101, HMDB51, and AVA-Kinetics for spatiotemporal localization.
Key Insights
Task-Specific Granularity
The research demonstrates that temporal granularity is highly task-specific. Fine-grained features are essential for boundary detection but can introduce "noise" in standard action recognition tasks. Conversely, persistent features provide the global context needed for general labels but are insensitive to precise temporal shifts.
Redefining "Temporal Awareness"
Interestingly, tasks like SSv2 and Diving48, which are typically thought to require fine-grained "temporal awareness" (e.g., distinguishing "opening" from "closing"), actually benefited more from persistent features. This suggests that "temporal awareness" in these contexts may rely more on global temporal context than on high-frequency frame-to-frame changes.
Superiority over Supervised Learning
TeG-FG (the fine-grained variant) outperformed supervised pre-training on Kinetics-GEBD by 8.9%. This indicates that manual labels can sometimes hinder a model's ability to learn the intrinsic temporal structure of video data.
Results and Performance
TeG achieved state-of-the-art results on several fronts:
- Diving48: Set a new record of 83.6% accuracy, surpassing all previous supervised and unsupervised methods.
- AVA-Kinetics: Achieved 28.7 mAP, an 8.9-point improvement over the supervised R3D-50 baseline.
- Efficiency: Reached 67.8% on Kinetics linear evaluation without the need for computationally expensive multi-clip sampling strategies.
Conclusion and Future Directions
TeG proves that a "one size fits all" approach to temporal representation is suboptimal. By balancing fine-grained and persistent features, the framework achieves broad success across varied video tasks. Future work aims to develop a "pyramid of representations" that can automatically adapt its temporal granularity to the specific requirements of a given task, removing the need for manual hyperparameter tuning.
Summary updated