Enhanced Motion-Text Alignment for Image-to-Video Transfer Learning

Wei Zhang, Chaoqun Wan, Tongliang Liu, Xinmei Tian, Xu Shen, Jieping Ye

Abstract

Adapting large-scale vision-language models like CLIP for video tasks often encounters a "motion gap," where models excel at spatial recognition but struggle with temporal dynamics. This paper introduces MoTED (MoTion-Enhanced Descriptions), a framework that bridges this gap by symmetrically aligning visual motion features with rich, LLM-generated textual descriptions. By leveraging GPT-4 to produce both conceptual and discriminative motion descriptions, MoTED provides the text encoder with the necessary nuance to distinguish between fine-grained actions. Combined with a dedicated visual motion encoder and a textual motion adapter, MoTED achieves state-of-the-art performance on motion-heavy benchmarks like Something-Something-V2 and demonstrates superior zero-shot generalization.

Introduction

The success of CLIP in image-level tasks has spurred significant interest in transferring its capabilities to video. However, standard adaptation techniques typically focus on the visual encoder—adding temporal layers—while leaving the text side unchanged. This creates a critical asymmetry: the visual representation becomes dynamic, but the textual representation remains anchored to static category names.

Consequently, models often fail to distinguish between actions that are spatially similar but temporally distinct, such as "clapping" versus "slapping." These single-word labels lack the nuance to describe how an action moves. Furthermore, CLIP’s text encoder has a strong bias toward spatial concepts (nouns) rather than temporal ones (verbs). MoTED aims to restore this symmetry by upgrading the textual representation to match the complexity of the visual motion.

The MoTED Framework

MoTED addresses the motion gap through three primary innovations designed to align dense visual motion with descriptive language.

1. Motion Description Generation

Instead of relying on simple category labels, MoTED utilizes Large Language Models (LLMs) like GPT-4 to generate two types of detailed descriptions:

  • Conceptual Motion Descriptions: General definitions of the motion concept (e.g., "a quick hand movement striking against something").
  • Discriminative Motion Descriptions: Prompts specifically designed to distinguish between easily confused classes (e.g., "How is slapping different from clapping?"). This provides the model with specific "features" to look for, such as the number of hands involved or the direction of motion.
Comparison of previous methods and the proposed MoTED framework, highlighting the use of conceptual and discriminative motion descriptions.

2. Symmetrical Architecture

The framework employs a dual-path approach for both vision and text to ensure features are properly disentangled and aligned.

  • Visual Motion Extraction: On the vision side, a frozen Image Encoder preserves spatial knowledge while a parallel, tunable Motion Encoder processes a denser stream of video frames. This encoder uses "Temporal Blocks" to capture both local pixel changes and global interactions across the video.
  • Textual Motion Adaptation: To handle the long-form GPT-generated descriptions, a Text Motion Adapter (using Multi-Head Self-Attention) is added to the text pipeline. This module learns to extract motion-specific semantics from the descriptions, overcoming the spatial bias of the original CLIP text encoder.
The MoTED architecture, showing the integration of the Image Encoder, Motion Encoder, Text Motion Adapter, and the LLM-driven description generation process.

Experimental Results

The authors evaluated MoTED on several key benchmarks, focusing on both supervised learning and zero-shot transfer.

  • Something-Something-V2 (SSv2): On this motion-heavy dataset, MoTED reached 71.9% Top-1 accuracy, outperforming previous leaders like DiST and ST-Adapter. An ablation study revealed that using discriminative descriptions alone boosted performance by 4.2% compared to standard category names.
  • Kinetics-400: The model achieved 86.2% Top-1 accuracy using a ViT-B/16 backbone.
  • Zero-Shot Generalization: MoTED showed remarkable transferability, improving Top-1 accuracy by +3.8% on HMDB51 and +6.0% on UCF101 over previous methods.

Conclusion

MoTED demonstrates that the "motion gap" in video transfer learning is as much a textual problem as a visual one. By providing the model with discriminative language that describes how actions differ, MoTED enables more precise alignment and robust recognition of complex temporal patterns. This symmetric approach ensures that the rich world-knowledge of image-language models is effectively translated into the dynamic domain of video.

Summary updated

Abstract

Extending large image-text pre-trained models (e.g., CLIP) for video understanding has made significant advancements. To enable the capability of CLIP to perceive dynamic information in videos, existing works are dedicated to equipping the visual encoder with various temporal modules. However, these methods exhibit “asymmetry” between the visual and textual sides, with neither temporal descriptions in input texts nor temporal modules in text encoder. This limitation hinders the potential of language supervision emphasized in CLIP, and restricts the learning of temporal features, as the text encoder has demonstrated limited proficiency in motion understanding. To address this issue, we propose leveraging “MoTion-Enhanced Descriptions” (MoTED) to facilitate the extraction of distinctive temporal features in videos. Specifically, we first generate discriminative motion-related descriptions via querying GPT-4 to compare easy-confusing action categories. Then, we incorporate both the visual and textual encoders with additional perception modules to process the video frames and generated descriptions, respectively. Finally, we adopt a contrastive loss to align the visual and textual motion features. Extensive experiments on five benchmarks show that MoTED surpasses state-of-the-art methods with convincing gaps, laying a solid foundation for empowering CLIP with strong temporal modeling.
BibTeX
@inproceedings{zhang2024enhanced,
  author = {Wei Zhang and Chaoqun Wan and Tongliang Liu and Xinmei Tian and Xu Shen and Jieping Ye},
  title = {{Enhanced Motion-Text Alignment for Image-to-Video Transfer Learning}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2024},
  doi = {10.1109/cvpr52733.2024.01751},
}
Figure