Abstract
Adapting image-based models like CLIP to video understanding typically requires extensive human-annotated data or complex self-supervised pretext tasks. We introduce Language-based Self-Supervision (LSS), a method that leverages the world knowledge of Large Language Models (LLMs) to create "Action Concept Spaces." By aligning video features with these linguistically structured spaces through a teacher-student framework, LSS achieves state-of-the-art performance in zero-shot and linear probing benchmarks without requiring manual video labels.
Introduction
The success of Contrastive Language-Image Pre-training (CLIP) has set a new standard for zero-shot image understanding. However, extending this capability to the temporal dimension of video remains a challenge. Existing video self-supervised learning (SSL) methods often rely on pretext tasks that may not preserve the broad semantic knowledge of CLIP. The primary hurdle is effective temporal modeling—distinguishing actions like "opening" versus "closing" a door—without the cost of manual annotation.
Methodology
The LSS framework addresses temporal modeling by forcing the model to operate within a linguistically structured space rather than an arbitrary feature space.
Framework Architecture
LSS utilizes a Vision Transformer (ViT) backbone modified with factorized space-time attention to process spatial and temporal information sequentially. The core of the method is the projection of video features into "Action Concept Spaces" defined by text generated from LLMs (e.g., GPT-3). These spaces consist of action categories and detailed visual descriptions of those actions.
Learning Objectives
The model is trained using two primary objectives:
- Concept Distillation (CD): A student model learns to match a teacher model's probability distribution across linguistic concepts.
- Concept Alignment (CA): This ensures that the visual representation of an action category aligns with the visual representation of its descriptive attributes. To prevent representation collapse, a Uniform Distribution Prior (UDP) is employed to encourage diverse concept utilization across the dataset.
Experimental Results
The model was pre-trained on Kinetics-400 (unlabeled) and evaluated across several benchmarks, including UCF-101, HMDB-51, and MSR-VTT.
Linear Probing and Zero-Shot Performance
LSS demonstrates significant improvements over baseline CLIP and other SSL methods. In linear probing on UCF-101, LSS achieved an accuracy of 91.1%, surpassing heavyweights such as VideoMAE and SVT.
In zero-shot transfer, LSS boosted CLIP’s performance on HMDB-51 to 51.4% (from a 47.2% baseline). Furthermore, in transductive zero-shot settings, it reached 55.0% on HMDB-51, establishing a new state-of-the-art for unsupervised video adaptation.
Discussion and Insights
LSS highlights the utility of LLMs as "world knowledge injectors," providing free supervision signals by describing what actions look like. By using language as a regularizer, the model becomes more interpretable, as its predictions are mapped directly to linguistic concepts. Crucially, LSS preserves and enhances the generality of the underlying CLIP model while successfully incorporating temporal awareness.
Limitations
Despite its strengths, LSS still faces challenges with fine-grained motion direction (e.g., distinguishing leftward vs. rightward movement) due to its reliance on image-based CLIP features. Additionally, the current model lacks object-level spatial awareness, suggesting that future work should integrate object-centric representations or pixel-level localization for more complex scene understanding.
Summary updated