Abstract
Self-supervised learning has made unsupervised pretraining relevant again for difficult computer vision tasks. The most effective self-supervised methods involve prediction tasks based on features extracted from diverse views of the data. DeepInfoMax (DIM) is a self-supervised method which leverages the internal structure of deep networks to construct such views, forming prediction tasks between local features which depend on small patches in an image and global features which depend on the whole image. In this paper, we extend DIM to the video domain by leveraging similar structure in spatio-temporal networks, producing a method we call Video Deep InfoMax (VDIM). We find that drawing views from both natural-rate sequences and temporally-downsampled sequences yields results on Kinetics-pretrained action recognition tasks which match or outperform prior state-of-the-art methods that use more costly large-time-scale transformer models. We also examine the effects of data augmentation and fine-tuning methods, accomplishing SoTA by a large margin when training only on the UCF-101 dataset.
Introduction
In computer vision, supervised learning requires massive amounts of human-annotated data, which is expensive, prone to error, and often fails to generalize to new tasks. This is especially true for video, where labeling actions is significantly more time-consuming than labeling static images. Self-supervised learning (SSL) offers a solution by deriving learning signals directly from the data's internal structure. While SSL has seen great success in static images, applying it to video is challenging due to the added temporal dimension. Previous high-performing video models often relied on massive "transformer" architectures. This research explores whether similar or better results can be achieved using more efficient, purely convolutional spatio-temporal networks by leveraging the principles of mutual information.
Methodology: The VDIM Framework
The core of the proposed approach is Video Deep InfoMax (VDIM), which builds on three main components to learn robust representations without human labels:
- The View Generator: A video clip is split into multiple subsequences. These "views" are transformed using random data augmentation (like cropping and color jittering) and temporal downsampling (e.g., skipping every other frame) to create diverse perspectives of the same underlying action.
- Mutual Information Maximization: The model is trained to recognize which "local" features (small spatio-temporal patches) belong to the same "global" video clip. It uses a contrastive loss (InfoNCE) to maximize the mutual information (MI) between these different views, forcing the encoder to learn representations that capture the underlying factors of the video rather than just pixel noise.
Network Architecture
To maintain efficiency, the model utilizes a spatio-temporal encoder based on the R(2+1)D architecture. Instead of using computationally heavy 3D convolutions, it decomposes the operation into 2D convolutions for spatial features followed by 1D convolutions for temporal features. This design makes the model more lightweight while maintaining the ability to capture complex motion patterns.
Key Insights and Results
The researchers evaluated VDIM on major action recognition benchmarks, including UCF-101, HMDB-51, and Kinetics 600. Several key insights emerged from the study:
- Local vs. Global Features: The model performs best when relating local features (limited view) to global features (full clip view). This forces the network to understand how small, localized movements contribute to a larger, coherent action.
- Temporal Downsampling: Drawing views from both natural-rate and temporally-downsampled sequences (effectively slow-motion vs. fast-motion) helps the model learn more robust temporal features.
- Efficiency vs. Performance: VDIM, using a standard convolutional backbone with roughly 17.3M parameters, matched or outperformed the Contrastive Bidirectional Transformer (CBT) on Kinetics-pretrained tasks, despite CBT being significantly more computationally expensive.
- State-of-the-Art Performance: VDIM achieved 74.1% accuracy on UCF-101 when trained only on that dataset, setting a new benchmark for models without external pretraining. When pretrained on Kinetics-600, it reached 79.7% on UCF-101 and 49.2% on HMDB-51.
Conclusion
Video Deep InfoMax demonstrates that maximizing mutual information across spatio-temporal views is a powerful paradigm for self-supervised video representation learning. By leveraging efficient R(2+1)D convolutions and clever view generation, VDIM challenges the necessity of expensive transformer models for action recognition tasks. Future work will explore more complex view operators and the impact of larger-scale hardware to further push the boundaries of unsupervised video understanding.
Summary updated