Language-based Action Concept Spaces Improve Video Self-Supervised Learning

Kanchana Ranasinghe, Michael S Ryoo

Abstract

Adapting image-based models like CLIP to video understanding typically requires extensive human-annotated data or complex self-supervised pretext tasks. We introduce Language-based Self-Supervision (LSS), a method that leverages the world knowledge of Large Language Models (LLMs) to create "Action Concept Spaces." By aligning video features with these linguistically structured spaces through a teacher-student framework, LSS achieves state-of-the-art performance in zero-shot and linear probing benchmarks without requiring manual video labels.

Introduction

The success of Contrastive Language-Image Pre-training (CLIP) has set a new standard for zero-shot image understanding. However, extending this capability to the temporal dimension of video remains a challenge. Existing video self-supervised learning (SSL) methods often rely on pretext tasks that may not preserve the broad semantic knowledge of CLIP. The primary hurdle is effective temporal modeling—distinguishing actions like "opening" versus "closing" a door—without the cost of manual annotation.

Methodology

The LSS framework addresses temporal modeling by forcing the model to operate within a linguistically structured space rather than an arbitrary feature space.

Framework Architecture

LSS utilizes a Vision Transformer (ViT) backbone modified with factorized space-time attention to process spatial and temporal information sequentially. The core of the method is the projection of video features into "Action Concept Spaces" defined by text generated from LLMs (e.g., GPT-3). These spaces consist of action categories and detailed visual descriptions of those actions.

Overview of the LSS framework showing the teacher-student architecture and dual learning objectives.

Learning Objectives

The model is trained using two primary objectives:

  1. Concept Distillation (CD): A student model learns to match a teacher model's probability distribution across linguistic concepts.
  2. Concept Alignment (CA): This ensures that the visual representation of an action category aligns with the visual representation of its descriptive attributes. To prevent representation collapse, a Uniform Distribution Prior (UDP) is employed to encourage diverse concept utilization across the dataset.

Experimental Results

The model was pre-trained on Kinetics-400 (unlabeled) and evaluated across several benchmarks, including UCF-101, HMDB-51, and MSR-VTT.

Linear Probing and Zero-Shot Performance

LSS demonstrates significant improvements over baseline CLIP and other SSL methods. In linear probing on UCF-101, LSS achieved an accuracy of 91.1%, surpassing heavyweights such as VideoMAE and SVT.

Linear probing performance of LSS compared to other state-of-the-art self-supervised methods.

In zero-shot transfer, LSS boosted CLIP’s performance on HMDB-51 to 51.4% (from a 47.2% baseline). Furthermore, in transductive zero-shot settings, it reached 55.0% on HMDB-51, establishing a new state-of-the-art for unsupervised video adaptation.

Discussion and Insights

LSS highlights the utility of LLMs as "world knowledge injectors," providing free supervision signals by describing what actions look like. By using language as a regularizer, the model becomes more interpretable, as its predictions are mapped directly to linguistic concepts. Crucially, LSS preserves and enhances the generality of the underlying CLIP model while successfully incorporating temporal awareness.

Limitations

Despite its strengths, LSS still faces challenges with fine-grained motion direction (e.g., distinguishing leftward vs. rightward movement) due to its reliance on image-based CLIP features. Additionally, the current model lacks object-level spatial awareness, suggesting that future work should integrate object-centric representations or pixel-level localization for more complex scene understanding.

Summary updated

Abstract

Recent contrastive language image pre-training has led to learning highly transferable and robust image representations. However, adapting these models to video domain with minimal supervision remains an open problem. We explore a simple step in that direction, using language tied self-supervised learning to adapt an image CLIP model to the video domain. A backbone modified for temporal modeling is trained under self-distillation settings with train objectives operating in an action concept space. Feature vectors of various action concepts extracted from a language encoder using relevant textual prompts construct this space. A large language model aware of actions and their attributes generates the relevant textual prompts. We introduce two train objectives, concept distillation and concept alignment, that retain generality of original representations while enforcing relations between actions and their attributes. Our approach improves zero-shot and linear probing performance on three action recognition benchmarks.
BibTeX
@inproceedings{ranasinghe2023language,
  author = {Kanchana Ranasinghe and Michael S Ryoo},
  title = {{Language-based Action Concept Spaces Improve Video Self-Supervised Learning}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2023},
  url = {https://openreview.net/forum?id=oyFyOPZUCs},
}
Figure