Abstract
Retrieving specific individuals from video sequences using natural language descriptions is a challenging task that extends beyond traditional image-based person re-identification. This paper introduces the Multi-Scale Structure Preservation (MSSP) framework, which leverages both appearance and action features to retrieve "person tubes"—sequences of bounding boxes—from video galleries. By enforcing structural constraints in a shared latent space, MSSP effectively aligns textual queries with multi-frame video segments. Experimental results on the newly introduced ActivityNet-PTRLD dataset and existing benchmarks demonstrate that MSSP significantly outperforms state-of-the-art methods.
Introduction
Traditional person re-identification (re-ID) systems typically rely on a query image to locate a person across different camera views. However, in many real-world scenarios, a visual query is unavailable, and search must be conducted using natural language descriptions, such as "a woman in a white tank top helping a boy ride a horse." While text-to-image search has seen progress, video-based retrieval introduces the complexity of temporal dynamics.
A "person tube"—defined as a sequence of bounding boxes tracking an individual through a video clip—contains critical information about actions and context that static images lack. The primary challenge is developing a joint embedding space where disparate modalities (text and video) can be compared directly.
Methodology: Multi-Scale Structure Preservation (MSSP)
The proposed MSSP framework addresses the retrieval task through a three-stage pipeline designed to capture both fine-grained appearance and temporal action dynamics.
Feature Extraction
The model utilizes a dual-stream approach for feature extraction:
- Appearance & Scene: A 2D ResNet extracts features from both the cropped person images within the bounding boxes and the full video frames to provide environmental context.
- Action Dynamics: A 3D ResNet processes 16-frame segments, termed "element-tubes," to capture the motion and behavior of the subject.
Multi-Scale Representation
To account for varying durations of actions, MSSP decomposes the person tube into element-tubes and subsequently aggregates them into "sub-tubes" of multiple scales. This hierarchical approach allows the model to recognize both instantaneous actions (e.g., a jump) and extended activities (e.g., running).
Joint Embedding and Loss Functions
To align text and video representations, the framework employs three structural constraints:
- Bidirectional Ranking Loss: Ensures that the correct description is ranked highest for a given tube and vice versa.
- Internal Structure Preservation: Encourages different sub-tubes and descriptions of the same individual to cluster together in the latent space.
- External Structure Preservation: Maximizes the distance between representations of different individuals to reduce retrieval ambiguity.
Experiments and Results
The authors evaluated MSSP on three datasets: ActivityNet-PTRLD (a new benchmark with 1.5 million bounding boxes for 6,068 persons), CUHK-PEDES (image-based person search), and MSR-VTT (general video retrieval).
Key findings include:
- Importance of Action: Integrating 3D action features improved Top-1 accuracy by over 10% compared to appearance-only models, highlighting the discriminative power of temporal information in video.
- State-of-the-Art Performance: On ActivityNet-PTRLD, MSSP achieved a Top-1 accuracy of 41.3% and a Top-10 accuracy of 89.3%, consistently outperforming baselines like DSPE and DSPE++.
- Cross-Modal Versatility: MSSP also surpassed previous state-of-the-art results on the CUHK-PEDES image dataset by 3.75% in Top-1 accuracy, demonstrating the robustness of the multi-scale structure preservation approach.
Conclusion
The MSSP framework provides a powerful solution for person tube retrieval via natural language. By combining 2D and 3D features with multi-scale structural constraints, the model effectively bridges the gap between text and video. Future work will focus on integrating real-time detection and tracking to create end-to-end retrieval systems capable of operating in unconstrained environments.
Summary updated