Abstract
Deep neural networks are efficient learning machines which leverage upon a large amount of manually labeled data for learning discriminative features. However, acquiring substantial amount of supervised data, especially for videos can be a tedious job across various computer vision tasks. This necessitates learning of visual features from videos in an unsupervised setting. In this paper, we propose a computationally simple, yet effective, framework to learn spatio-temporal feature embedding from unlabeled videos. We train a Convolutional 3D Siamese network using positive and negative pairs mined from videos under certain probabilistic assumptions. Experimental results on three datasets demonstrate that our proposed framework is able to learn weights which can be used for same as well as cross dataset and tasks.
Introduction
Deep learning's success is heavily dependent on the availability of massive labeled datasets. While image annotation is challenging, video annotation—which requires marking actions, objects, and temporal boundaries—is exponentially more difficult and expensive. This creates a significant bottleneck for scaling video-based AI.
The core challenge in unsupervised video learning is defining a "tractable task"—a proxy goal that forces a network to learn meaningful spatio-temporal features without explicit labels. Previous methods, such as Autoencoders or temporal ordering tasks, often suffer from high computational costs or fail to capture high-level semantic structures. This research introduces a framework designed for both semantic richness and computational scalability.
Proposed Framework
The authors propose an Unsupervised 3D Siamese Network architecture. The core of the system is a Convolutional 3D (C3D) backbone that processes 16-frame spatio-temporal volumes.
The framework relies on a novel Pair Mining Strategy:
- Positive Pairs: Based on the hypothesis of visual continuity, two clips from the same video within a certain spatial and temporal proximity are likely to share semantic content. For example, a soccer ball in one part of a frame is likely part of the same "soccer" action in a nearby spatio-temporal region. To ensure the network learns features rather than simple pixel matching, random color transformations and horizontal flips are applied.
- Negative Pairs: Clips are pulled from entirely different videos, leveraging the statistical likelihood that random videos do not share the same specific action.
- Training: The Siamese network is trained using a Hinge Loss function to project clips into a feature space where related clips are clustered and unrelated ones are separated.
Experimental Setup
The framework was evaluated on three major benchmarks:
- UCF101: Used for initial unsupervised training and action recognition evaluation.
- HMDB51: A smaller dataset used to evaluate the transferability of learned features through fine-tuning.
- ASLAN (Action Similarity Labeling): Used to test the model's ability to judge action similarity on unseen classes.
Results and Discussion
The proposed method demonstrates significant improvements in both performance and efficiency. On HMDB51, unsupervised pre-training achieved 20.6% accuracy, outperforming random initialization (19.2%) and other unsupervised baselines. On ASLAN, the model reached 63.5% accuracy, a notable jump from the 57.3% baseline.
A key advantage is scalability. By avoiding expensive Optical Flow calculations and using raw RGB frames with randomized mining, the data preparation is over 4,000 times faster than flow-based methods ($1.8 \times 10^{-5}$ vs $8.0 \times 10^{-2}$ seconds per pair). Furthermore, visualizations of the "pool5" layer indicate that the network naturally learns to focus on human-object interactions and motion-heavy regions without any supervised guidance.
Conclusion
This work presents a scalable approach to unsupervised feature learning that leverages the inherent semantic consistency of natural videos. While a performance gap still exists compared to fully supervised state-of-the-art models, the efficiency and transferability of these features provide a strong foundation for large-scale video analysis. Future work may explore deeper architectures and more sophisticated data augmentations to further bridge the supervision gap.
Summary updated