Abstract
Text-to-Video Person Retrieval (TVPR) is an emerging computer vision task that extends traditional person re-identification (ReID) into the temporal domain. Unlike image-based retrieval, TVPR utilizes natural language descriptions to identify specific individuals within video sequences. This paper introduces TVPReid, the first large-scale benchmark for this task, and proposes the Multielement Feature Guided Fragments Learning (MFGF) framework. By integrating spatial appearance with motion dynamics, MFGF effectively addresses challenges such as temporary occlusion and visual blur, setting a new state-of-the-art for video-based person retrieval.
Introduction
Traditional text-based person retrieval relies on matching linguistic descriptions to static images. However, real-world surveillance scenarios often involve dynamic complexities that a single frame cannot capture. A target individual might be momentarily obscured by an object or identified primarily through a specific gait or sequence of actions.
As shown in Figure 1, image-based models struggle when a person is partially blocked (e.g., by an umbrella). The core insight of this work is that video provides temporal redundancy; if an attribute is hidden in one frame, it may be visible in another. To facilitate research in this area, we present the TVPReid dataset and a novel dual-stream learning strategy.
The TVPReid Benchmark
To address the lack of specialized data, we introduced the TVPReid dataset. This benchmark re-annotates three classic video ReID datasets—PRID-2011, iLIDS-VID, and DukeMTMC-VideoReID—with detailed natural language descriptions.
- Scale: 6,559 unique pedestrian videos and 13,118 descriptive sentences.
- Granularity: Descriptions average 30 words, covering physical appearance, specific actions, and environmental interactions.
Multielement Feature Guided Fragments Learning (MFGF)
The MFGF framework decomposes the retrieval task into manageable "fragments" across text and video modalities.
Fragments Learning and Motion Encoding
The model employs a dual-stream video encoder. A Visual Encoder (ViT) captures spatial details, while a Motion Encoder (S3D) utilizes 3D convolutions to extract temporal patterns. This allows the model to understand not just what a person looks like, but how they move through time.
Dual-Space Learning and "Tips"
To bridge the cross-modal gap, MFGF utilizes a Text Prompter to extract high-value keywords (verbs, nouns, adjectives) into a "Vocab Bank." These keywords, or "Tips," guide the model in two distinct spaces:
- Common Space: Aligns global text and video features via contrastive learning.
- Dual-Distilled ($D^2$) Space: Filters out redundant information (e.g., background noise or linguistic filler) to focus exclusively on the target person's unique traits.
Experimental Results
MFGF was evaluated against state-of-the-art video retrieval methods adapted for the TVPR task. Our framework consistently outperformed existing models across all sub-datasets, demonstrating the importance of motion as a biometric cue.
| Dataset | Metric | X-pool | Ours (MFGF) |
|---|---|---|---|
| TVPReid-PRID | R@1 | 21.1% | 32.3% |
| TVPReid-iLIDs | R@1 | 14.7% | 30.0% |
| TVPReid-Duke | R@1 | 25.2% | 35.2% |
Notably, on the PRID sub-dataset, the model achieved a 100% R@50 score, ensuring the target video was always present within the top 50 results.
Conclusion and Future Work
TVPR represents a significant advancement in person retrieval, moving toward more robust, context-aware surveillance systems. While MFGF shows strong performance, future work will focus on reducing the computational complexity of 3D convolutions and exploring the integration of audio cues to further enhance retrieval accuracy in noisy environments.
Summary updated