Introduction
Text-based Person Search (TPS) involves identifying a specific individual within large-scale, uncropped scene images using only natural language descriptions. Unlike standard person retrieval, which operates on pre-cropped images, TPS requires simultaneous person detection and identity matching. This task faces two primary challenges: the scarcity of high-quality, multi-description datasets and the inherent conflict between detection (finding any human) and retrieval (finding a specific identity).
The PRW-TPS-CN Dataset
To address data scarcity, this work introduces PRW-TPS-CN, a large-scale bilingual dataset. It contains 47,102 structured descriptions (in both Chinese and English) for 32,206 bounding boxes across 11,776 scene images. To minimize annotator bias, each training image includes at least two descriptions. Furthermore, the descriptions follow a "top-to-bottom" structure (hair $\rightarrow$ clothing $\rightarrow$ shoes), which helps models learn spatial-textual correlations more effectively.
Text Prototype Attention Network (TPAN)
The proposed Text Prototype Attention Network (TPAN) mitigates the "feature ambiguity" problem by using aggregated textual information to guide visual attention.
Prototype Table and Guided Attention
Instead of relying on a single, potentially noisy text query, TPAN maintains a Prototype Table. During training, features from multiple descriptions of the same identity are aggregated into a "prototype"—a stable, canonical representation of that person's appearance.
As shown in the architecture, these prototypes are compared against visual features using cosine similarity to generate a Correspond Map. This map serves as a guide for the Attention Module, forcing the visual backbone to focus on specific attributes (e.g., a red jacket or a black backpack) rather than just general human features. This alignment ensures that the detector produces crops optimized for the subsequent retrieval stage.
Experimental Results
TPAN sets a new state-of-the-art on the PRW-TPS-CN dataset, achieving 29.00% mAP and 46.63% Rank-1 accuracy. This significantly outperforms previous methods like SDPG (11.93% mAP). Using a BERT-based text encoder and a ResNet-101 visual backbone further improves performance to 30.30% mAP. Qualitative analysis via Class Activation Maps confirms that TPAN accurately focuses on descriptive keywords, bridging the gap between detection and fine-grained retrieval.
Conclusion
By introducing the rich PRW-TPS-CN dataset and the prototype-guided TPAN architecture, this research provides a robust framework for bilingual text-based person search. The use of identity prototypes as "anchors" effectively resolves the conflict between detection and retrieval, paving the way for more accurate real-world search applications.
Summary updated