Abstract
Adapting large-scale vision-language models like CLIP for video tasks often encounters a "motion gap," where models excel at spatial recognition but struggle with temporal dynamics. This paper introduces MoTED (MoTion-Enhanced Descriptions), a framework that bridges this gap by symmetrically aligning visual motion features with rich, LLM-generated textual descriptions. By leveraging GPT-4 to produce both conceptual and discriminative motion descriptions, MoTED provides the text encoder with the necessary nuance to distinguish between fine-grained actions. Combined with a dedicated visual motion encoder and a textual motion adapter, MoTED achieves state-of-the-art performance on motion-heavy benchmarks like Something-Something-V2 and demonstrates superior zero-shot generalization.
Introduction
The success of CLIP in image-level tasks has spurred significant interest in transferring its capabilities to video. However, standard adaptation techniques typically focus on the visual encoder—adding temporal layers—while leaving the text side unchanged. This creates a critical asymmetry: the visual representation becomes dynamic, but the textual representation remains anchored to static category names.
Consequently, models often fail to distinguish between actions that are spatially similar but temporally distinct, such as "clapping" versus "slapping." These single-word labels lack the nuance to describe how an action moves. Furthermore, CLIP’s text encoder has a strong bias toward spatial concepts (nouns) rather than temporal ones (verbs). MoTED aims to restore this symmetry by upgrading the textual representation to match the complexity of the visual motion.
The MoTED Framework
MoTED addresses the motion gap through three primary innovations designed to align dense visual motion with descriptive language.
1. Motion Description Generation
Instead of relying on simple category labels, MoTED utilizes Large Language Models (LLMs) like GPT-4 to generate two types of detailed descriptions:
- Conceptual Motion Descriptions: General definitions of the motion concept (e.g., "a quick hand movement striking against something").
- Discriminative Motion Descriptions: Prompts specifically designed to distinguish between easily confused classes (e.g., "How is slapping different from clapping?"). This provides the model with specific "features" to look for, such as the number of hands involved or the direction of motion.
2. Symmetrical Architecture
The framework employs a dual-path approach for both vision and text to ensure features are properly disentangled and aligned.
- Visual Motion Extraction: On the vision side, a frozen Image Encoder preserves spatial knowledge while a parallel, tunable Motion Encoder processes a denser stream of video frames. This encoder uses "Temporal Blocks" to capture both local pixel changes and global interactions across the video.
- Textual Motion Adaptation: To handle the long-form GPT-generated descriptions, a Text Motion Adapter (using Multi-Head Self-Attention) is added to the text pipeline. This module learns to extract motion-specific semantics from the descriptions, overcoming the spatial bias of the original CLIP text encoder.
Experimental Results
The authors evaluated MoTED on several key benchmarks, focusing on both supervised learning and zero-shot transfer.
- Something-Something-V2 (SSv2): On this motion-heavy dataset, MoTED reached 71.9% Top-1 accuracy, outperforming previous leaders like DiST and ST-Adapter. An ablation study revealed that using discriminative descriptions alone boosted performance by 4.2% compared to standard category names.
- Kinetics-400: The model achieved 86.2% Top-1 accuracy using a ViT-B/16 backbone.
- Zero-Shot Generalization: MoTED showed remarkable transferability, improving Top-1 accuracy by +3.8% on HMDB51 and +6.0% on UCF101 over previous methods.
Conclusion
MoTED demonstrates that the "motion gap" in video transfer learning is as much a textual problem as a visual one. By providing the model with discriminative language that describes how actions differ, MoTED enables more precise alignment and robust recognition of complex temporal patterns. This symmetric approach ensures that the rich world-knowledge of image-language models is effectively translated into the dynamic domain of video.
Summary updated