PIDRo: Parallel Isomeric Attention with Dynamic Routing for Text-Video Retrieval

Peiyan Guan, Renjing Pei, Bin Shao, Jianzhuang Liu, Weimian Li, Jiaxi Gu, Hang Xu, Songcen Xu, Youliang Yan, Edmund Y. Lam

Introduction

Large-scale pre-trained models like CLIP have significantly advanced image-text tasks, yet their adaptation to video retrieval remains a challenge. Current methods often struggle with two primary issues: the Temporal Gap and the Granularity Gap. The temporal gap arises because CLIP is optimized for static images, often failing to capture the subtle, fine-grained temporal dynamics essential for video understanding. The granularity gap occurs because standard CLIP encoders typically collapse information into a single global token, losing the specific word-level or patch-level details necessary for precise matching.

To address these challenges, we present PIDRo (Parallel Isomeric Attention with Dynamic Routing), a framework that introduces architectural innovations to bridge these gaps while preserving the rich knowledge of pre-trained CLIP models.

Methodology

PIDRo introduces three core components to enhance text-video alignment:

Parallel Isomeric Attention (Video Encoder)

Inspired by chemical isomers—molecules with identical atoms but different structures—this module employs two parallel branches to process video data:

  • S-T (Spatial-Temporal) Branch: This branch processes spatial information before temporal, focusing on high-level frame representations.
  • T-S (Temporal-Spatial) Branch: This branch prioritizes temporal information within localized "patch cubes" before spatial processing, capturing fine-grained motion that frame-level analysis might miss.

Dynamic Routing (Text Encoder)

Standard CLIP text encoders often bottleneck information into a global [SEP] token. PIDRo’s Dynamic Routing module "unpacks" this global summary, using an MLP to redistribute relevant knowledge back to individual word tokens. This ensures that specific descriptors (e.g., "frying," "baseball") retain their distinctiveness for matching.

The PIDRo architecture featuring Parallel Isomeric Attention and Dynamic Routing.

Token-wise Interaction and Efficiency

PIDRo calculates similarity scores between individual words and visual elements (frames and patches). To maintain computational efficiency, a Patch Token Selector is utilized in the T-S branch to focus only on the $K$ most informative patches, reducing the overhead of the dual-branch architecture.

Experimental Results

PIDRo was evaluated on five major benchmarks: MSR-VTT, MSVD, LSMDC, DiDeMo, and ActivityNet. The model achieved state-of-the-art (SOTA) performance across all datasets. Notably, on MSR-VTT, PIDRo reached a Recall@1 (R@1) of 48.2%, which increased to 55.9% when utilizing a ViT-B/16 backbone and Dual Softmax (DSL) post-processing.

Qualitative Analysis

Visualization of the attention weights confirms that PIDRo successfully aligns specific words with relevant visual regions and frames. As shown in the figure below, the dynamic routing mechanism allows the model to correctly associate actions and objects with their corresponding temporal and spatial locations in the video.

Visualization of token-wise interaction and dynamic routing results showing fine-grained alignment between words and video segments.

Conclusion

PIDRo effectively adapts CLIP for video retrieval by introducing parallel isomeric attention and dynamic routing. This dual-branch approach captures both global context and fine-grained temporal dynamics without altering CLIP's internal weights. While the dual-branch architecture adds some complexity, the use of patch selection ensures it remains efficient. Future work will explore extending this fine-grained attention to long-form video content and unifying the training pipeline.

Summary updated

Abstract

Text-video retrieval is a fundamental task with high practical value in multi-modal research. Inspired by the great success of pre-trained image-text models with large-scale data, such as CLIP, many methods are proposed to transfer the strong representation learning capability of CLIP to text-video retrieval. However, due to the modality difference between videos and images, how to effectively adapt CLIP to the video domain is still underexplored. In this paper, we investigate this problem from two aspects. First, we enhance the transferred image encoder of CLIP for fine-grained video understanding in a seamless fashion. Second, we conduct fine-grained contrast between videos and texts from both model improvement and loss design. Particularly, we propose a fine-grained contrastive model equipped with parallel isomeric attention and dynamic routing, namely PIDRo, for text-video retrieval. The parallel isomeric attention module is used as the video encoder, which consists of two parallel branches modeling the spatial-temporal information of videos from both patch and frame levels. The dynamic routing module is constructed to enhance the text encoder of CLIP, generating informative word representations by distributing the fine-grained information to the related word tokens within a sentence. Such model design provides us with informative patch, frame and word representations. We then conduct token-wise interaction upon them. With the enhanced encoders and the token-wise loss, we are able to achieve finer-grained text-video alignment and more accurate retrieval. PIDRo obtains state-of-the-art performance over various text-video retrieval benchmarks, including MSR-VTT, MSVD, LSMDC, DiDeMo and ActivityNet.
BibTeX
@inproceedings{guan2023pidro,
  author = {Peiyan Guan and Renjing Pei and Bin Shao and Jianzhuang Liu and Weimian Li and Jiaxi Gu and Hang Xu and Songcen Xu and Youliang Yan and Edmund Y. Lam},
  title = {{PIDRo: Parallel Isomeric Attention with Dynamic Routing for Text-Video Retrieval}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
  doi = {10.1109/iccv51070.2023.01025},
}
Figure