TVPR: Text-to-Video Person Retrieval and a New Benchmark

Fan Ni, Xu Zhang, Jianhui Wu, Guannan Dong, Aichun Zhu, Hui Liu, Yue ZhangarXiv 2023 · DOI · arXiv

Abstract

Text-to-Video Person Retrieval (TVPR) is an emerging computer vision task that extends traditional person re-identification (ReID) into the temporal domain. Unlike image-based retrieval, TVPR utilizes natural language descriptions to identify specific individuals within video sequences. This paper introduces TVPReid, the first large-scale benchmark for this task, and proposes the Multielement Feature Guided Fragments Learning (MFGF) framework. By integrating spatial appearance with motion dynamics, MFGF effectively addresses challenges such as temporary occlusion and visual blur, setting a new state-of-the-art for video-based person retrieval.

Introduction

Traditional text-based person retrieval relies on matching linguistic descriptions to static images. However, real-world surveillance scenarios often involve dynamic complexities that a single frame cannot capture. A target individual might be momentarily obscured by an object or identified primarily through a specific gait or sequence of actions.

Figure 1: Comparison between image-based (a) and video-based (b) person retrieval under occlusion. While static images fail when key attributes are hidden, video provides temporal context to maintain identity.

As shown in Figure 1, image-based models struggle when a person is partially blocked (e.g., by an umbrella). The core insight of this work is that video provides temporal redundancy; if an attribute is hidden in one frame, it may be visible in another. To facilitate research in this area, we present the TVPReid dataset and a novel dual-stream learning strategy.

The TVPReid Benchmark

To address the lack of specialized data, we introduced the TVPReid dataset. This benchmark re-annotates three classic video ReID datasets—PRID-2011, iLIDS-VID, and DukeMTMC-VideoReID—with detailed natural language descriptions.

  • Scale: 6,559 unique pedestrian videos and 13,118 descriptive sentences.
  • Granularity: Descriptions average 30 words, covering physical appearance, specific actions, and environmental interactions.

Multielement Feature Guided Fragments Learning (MFGF)

The MFGF framework decomposes the retrieval task into manageable "fragments" across text and video modalities.

Fragments Learning and Motion Encoding

The model employs a dual-stream video encoder. A Visual Encoder (ViT) captures spatial details, while a Motion Encoder (S3D) utilizes 3D convolutions to extract temporal patterns. This allows the model to understand not just what a person looks like, but how they move through time.

Figure 2: Mapping natural language queries to temporal sequences and motion details in video. The model aligns specific textual actions (1, 2, 3) with corresponding video segments.

Dual-Space Learning and "Tips"

To bridge the cross-modal gap, MFGF utilizes a Text Prompter to extract high-value keywords (verbs, nouns, adjectives) into a "Vocab Bank." These keywords, or "Tips," guide the model in two distinct spaces:

  1. Common Space: Aligns global text and video features via contrastive learning.
  2. Dual-Distilled ($D^2$) Space: Filters out redundant information (e.g., background noise or linguistic filler) to focus exclusively on the target person's unique traits.

Experimental Results

MFGF was evaluated against state-of-the-art video retrieval methods adapted for the TVPR task. Our framework consistently outperformed existing models across all sub-datasets, demonstrating the importance of motion as a biometric cue.

Dataset Metric X-pool Ours (MFGF)
TVPReid-PRID R@1 21.1% 32.3%
TVPReid-iLIDs R@1 14.7% 30.0%
TVPReid-Duke R@1 25.2% 35.2%

Notably, on the PRID sub-dataset, the model achieved a 100% R@50 score, ensuring the target video was always present within the top 50 results.

Conclusion and Future Work

TVPR represents a significant advancement in person retrieval, moving toward more robust, context-aware surveillance systems. While MFGF shows strong performance, future work will focus on reducing the computational complexity of 3D convolutions and exploring the integration of audio cues to further enhance retrieval accuracy in noisy environments.

Summary updated

BibTeX
@misc{ni2023tvpr,
  author = {Fan Ni and Xu Zhang and Jianhui Wu and Guannan Dong and Aichun Zhu and Hui Liu and Yue Zhang},
  title = {{TVPR: Text-to-Video Person Retrieval and a New Benchmark}},
  year = {2023},
  eprint = {2307.07184},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2307.07184},
}
Figure