Prototype-Guided Text-based Person Search based on Rich Chinese Descriptions

Ziqiang Wu, Bingpeng MaarXiv 2023 · DOI · arXiv

Introduction

Text-based Person Search (TPS) involves identifying a specific individual within large-scale, uncropped scene images using only natural language descriptions. Unlike standard person retrieval, which operates on pre-cropped images, TPS requires simultaneous person detection and identity matching. This task faces two primary challenges: the scarcity of high-quality, multi-description datasets and the inherent conflict between detection (finding any human) and retrieval (finding a specific identity).

Text-based Person Search involves locating a specific individual in a complex scene based on a textual query.

The PRW-TPS-CN Dataset

To address data scarcity, this work introduces PRW-TPS-CN, a large-scale bilingual dataset. It contains 47,102 structured descriptions (in both Chinese and English) for 32,206 bounding boxes across 11,776 scene images. To minimize annotator bias, each training image includes at least two descriptions. Furthermore, the descriptions follow a "top-to-bottom" structure (hair $\rightarrow$ clothing $\rightarrow$ shoes), which helps models learn spatial-textual correlations more effectively.

Text Prototype Attention Network (TPAN)

The proposed Text Prototype Attention Network (TPAN) mitigates the "feature ambiguity" problem by using aggregated textual information to guide visual attention.

Prototype Table and Guided Attention

Instead of relying on a single, potentially noisy text query, TPAN maintains a Prototype Table. During training, features from multiple descriptions of the same identity are aggregated into a "prototype"—a stable, canonical representation of that person's appearance.

The TPAN architecture uses a Prototype Table to generate attention maps that guide the visual backbone toward identity-salient regions.

As shown in the architecture, these prototypes are compared against visual features using cosine similarity to generate a Correspond Map. This map serves as a guide for the Attention Module, forcing the visual backbone to focus on specific attributes (e.g., a red jacket or a black backpack) rather than just general human features. This alignment ensures that the detector produces crops optimized for the subsequent retrieval stage.

Experimental Results

TPAN sets a new state-of-the-art on the PRW-TPS-CN dataset, achieving 29.00% mAP and 46.63% Rank-1 accuracy. This significantly outperforms previous methods like SDPG (11.93% mAP). Using a BERT-based text encoder and a ResNet-101 visual backbone further improves performance to 30.30% mAP. Qualitative analysis via Class Activation Maps confirms that TPAN accurately focuses on descriptive keywords, bridging the gap between detection and fine-grained retrieval.

Conclusion

By introducing the rich PRW-TPS-CN dataset and the prototype-guided TPAN architecture, this research provides a robust framework for bilingual text-based person search. The use of identity prototypes as "anchors" effectively resolves the conflict between detection and retrieval, paving the way for more accurate real-world search applications.

Summary updated

Abstract

Text-based person search aims to simultaneously localize and identify the target person based on query text from uncropped scene images, which can be regarded as the unified task of person detection and text-based person retrieval task. In this work, we propose a large-scale benchmark dataset named PRW-TPS-CN based on the widely used person search dataset PRW. Our dataset contains 47,102 sentences, which means there is quite more information than existing dataset. These texts precisely describe the person images from top to bottom, which in line with the natural description order. We also provide both Chinese and English descriptions in our dataset for more comprehensive evaluation. These characteristics make our dataset more applicable. To alleviate the inconsistency between person detection and text-based person retrieval, we take advantage of the rich texts in PRW-TPS-CN dataset. We propose to aggregate multiple texts as text prototypes to maintain the prominent text features of a person, which can better reflect the whole character of a person. The overall prototypes lead to generating the image attention map to eliminate the detection misalignment causing the decrease of text-based person retrieval. Thus, the inconsistency between person detection and text-based person retrieval is largely alleviated. We conduct extensive experiments on the PRW-TPS-CN dataset. The experimental results show the PRW-TPS-CN dataset's effectiveness and the state-of-the-art performance of our approach.
BibTeX
@misc{wu2023prototype,
  author = {Ziqiang Wu and Bingpeng Ma},
  title = {{Prototype-Guided Text-based Person Search based on Rich Chinese Descriptions}},
  year = {2023},
  eprint = {2312.14834},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2312.14834},
}
Figure