Introduction
Large-scale pre-trained models like CLIP have significantly advanced image-text tasks, yet their adaptation to video retrieval remains a challenge. Current methods often struggle with two primary issues: the Temporal Gap and the Granularity Gap. The temporal gap arises because CLIP is optimized for static images, often failing to capture the subtle, fine-grained temporal dynamics essential for video understanding. The granularity gap occurs because standard CLIP encoders typically collapse information into a single global token, losing the specific word-level or patch-level details necessary for precise matching.
To address these challenges, we present PIDRo (Parallel Isomeric Attention with Dynamic Routing), a framework that introduces architectural innovations to bridge these gaps while preserving the rich knowledge of pre-trained CLIP models.
Methodology
PIDRo introduces three core components to enhance text-video alignment:
Parallel Isomeric Attention (Video Encoder)
Inspired by chemical isomers—molecules with identical atoms but different structures—this module employs two parallel branches to process video data:
- S-T (Spatial-Temporal) Branch: This branch processes spatial information before temporal, focusing on high-level frame representations.
- T-S (Temporal-Spatial) Branch: This branch prioritizes temporal information within localized "patch cubes" before spatial processing, capturing fine-grained motion that frame-level analysis might miss.
Dynamic Routing (Text Encoder)
Standard CLIP text encoders often bottleneck information into a global [SEP] token. PIDRo’s Dynamic Routing module "unpacks" this global summary, using an MLP to redistribute relevant knowledge back to individual word tokens. This ensures that specific descriptors (e.g., "frying," "baseball") retain their distinctiveness for matching.
Token-wise Interaction and Efficiency
PIDRo calculates similarity scores between individual words and visual elements (frames and patches). To maintain computational efficiency, a Patch Token Selector is utilized in the T-S branch to focus only on the $K$ most informative patches, reducing the overhead of the dual-branch architecture.
Experimental Results
PIDRo was evaluated on five major benchmarks: MSR-VTT, MSVD, LSMDC, DiDeMo, and ActivityNet. The model achieved state-of-the-art (SOTA) performance across all datasets. Notably, on MSR-VTT, PIDRo reached a Recall@1 (R@1) of 48.2%, which increased to 55.9% when utilizing a ViT-B/16 backbone and Dual Softmax (DSL) post-processing.
Qualitative Analysis
Visualization of the attention weights confirms that PIDRo successfully aligns specific words with relevant visual regions and frames. As shown in the figure below, the dynamic routing mechanism allows the model to correctly associate actions and objects with their corresponding temporal and spatial locations in the video.
Conclusion
PIDRo effectively adapts CLIP for video retrieval by introducing parallel isomeric attention and dynamic routing. This dual-branch approach captures both global context and fine-grained temporal dynamics without altering CLIP's internal weights. While the dual-branch architecture adds some complexity, the use of patch selection ensures it remains efficient. Future work will explore extending this fine-grained attention to long-form video content and unifying the training pipeline.
Summary updated