Person Tube Retrieval via Language Description

Hehe Fan, Yi Yang

Abstract

Retrieving specific individuals from video sequences using natural language descriptions is a challenging task that extends beyond traditional image-based person re-identification. This paper introduces the Multi-Scale Structure Preservation (MSSP) framework, which leverages both appearance and action features to retrieve "person tubes"—sequences of bounding boxes—from video galleries. By enforcing structural constraints in a shared latent space, MSSP effectively aligns textual queries with multi-frame video segments. Experimental results on the newly introduced ActivityNet-PTRLD dataset and existing benchmarks demonstrate that MSSP significantly outperforms state-of-the-art methods.

Introduction

Traditional person re-identification (re-ID) systems typically rely on a query image to locate a person across different camera views. However, in many real-world scenarios, a visual query is unavailable, and search must be conducted using natural language descriptions, such as "a woman in a white tank top helping a boy ride a horse." While text-to-image search has seen progress, video-based retrieval introduces the complexity of temporal dynamics.

A "person tube"—defined as a sequence of bounding boxes tracking an individual through a video clip—contains critical information about actions and context that static images lack. The primary challenge is developing a joint embedding space where disparate modalities (text and video) can be compared directly.

Overview of the person tube retrieval task using natural language queries.

Methodology: Multi-Scale Structure Preservation (MSSP)

The proposed MSSP framework addresses the retrieval task through a three-stage pipeline designed to capture both fine-grained appearance and temporal action dynamics.

Feature Extraction

The model utilizes a dual-stream approach for feature extraction:

  1. Appearance & Scene: A 2D ResNet extracts features from both the cropped person images within the bounding boxes and the full video frames to provide environmental context.
  2. Action Dynamics: A 3D ResNet processes 16-frame segments, termed "element-tubes," to capture the motion and behavior of the subject.

Multi-Scale Representation

To account for varying durations of actions, MSSP decomposes the person tube into element-tubes and subsequently aggregates them into "sub-tubes" of multiple scales. This hierarchical approach allows the model to recognize both instantaneous actions (e.g., a jump) and extended activities (e.g., running).

Multi-scale representation of person tubes, showing cropped person images and full video frames.

Joint Embedding and Loss Functions

To align text and video representations, the framework employs three structural constraints:

  • Bidirectional Ranking Loss: Ensures that the correct description is ranked highest for a given tube and vice versa.
  • Internal Structure Preservation: Encourages different sub-tubes and descriptions of the same individual to cluster together in the latent space.
  • External Structure Preservation: Maximizes the distance between representations of different individuals to reduce retrieval ambiguity.

Experiments and Results

The authors evaluated MSSP on three datasets: ActivityNet-PTRLD (a new benchmark with 1.5 million bounding boxes for 6,068 persons), CUHK-PEDES (image-based person search), and MSR-VTT (general video retrieval).

Key findings include:

  • Importance of Action: Integrating 3D action features improved Top-1 accuracy by over 10% compared to appearance-only models, highlighting the discriminative power of temporal information in video.
  • State-of-the-Art Performance: On ActivityNet-PTRLD, MSSP achieved a Top-1 accuracy of 41.3% and a Top-10 accuracy of 89.3%, consistently outperforming baselines like DSPE and DSPE++.
  • Cross-Modal Versatility: MSSP also surpassed previous state-of-the-art results on the CUHK-PEDES image dataset by 3.75% in Top-1 accuracy, demonstrating the robustness of the multi-scale structure preservation approach.

Conclusion

The MSSP framework provides a powerful solution for person tube retrieval via natural language. By combining 2D and 3D features with multi-scale structural constraints, the model effectively bridges the gap between text and video. Future work will focus on integrating real-time detection and tracking to create end-to-end retrieval systems capable of operating in unconstrained environments.

Summary updated

Abstract

This paper focuses on the problem of person tube (a sequence of bounding boxes which encloses a person in a video) retrieval using a natural language query. Different from images in person re-identification (re-ID) or person search, besides appearance, person tube contains abundant action and information. We exploit a 2D and a 3D residual networks (ResNets) to extract the appearance and action representation, respectively. To transform tubes and descriptions into a shared latent space where data from the two different modalities can be compared directly, we propose a Multi-Scale Structure Preservation (MSSP) approach. MSSP splits a person tube into several element-tubes on average, whose features are extracted by the two ResNets. Any number of consecutive element-tubes forms a sub-tube. MSSP considers the following constraints for sub-tubes and descriptions in the shared space. 1) Bidirectional ranking. Matching sub-tubes (resp. descriptions) should get ranked higher than incorrect ones for each description (resp. sub-tube). 2) External structure preservation. Sub-tubes (resp. descriptions) from different persons should stay away from each other. 3) Internal structure preservation. Sub-tubes (resp. descriptions) from the same person should be close to each other. Experimental results on person tube retrieval via language description and other two related tasks demonstrate the efficacy of MSSP.
BibTeX
@inproceedings{fan2020person,
  author = {Hehe Fan and Yi Yang},
  title = {{Person Tube Retrieval via Language Description}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.1609/aaai.v34i07.6704},
}
Figure