Abstract
Person search has drawn increasing attention due to its real-world applications and research significance. It aims to find a probe person in a gallery of scene images, with applications ranging from criminal investigation to missing person search. This paper surveys recent works on image-based and text-based person search, focusing on three significant challenges: discriminative person features, the query-person gap, and detection-identification inconsistency. We summarize evaluation results and discuss promising future research directions.
Introduction
Traditional Person Re-identification (Re-ID) typically assumes the availability of pre-cropped person images. However, real-world scenarios require finding individuals within unconstrained, cluttered scene images. Person Search addresses this by combining Person Detection and Person Identification into a unified task. This evolution introduces a fundamental conflict: detection models prioritize common human features, while identification models require unique, discriminative features.
Methodological Frameworks
Person search is primarily categorized by the nature of the query: image-based or text-based.
Image-based Person Search uses a cropped image of the target as a query. Modern frameworks often utilize a shared backbone to extract feature maps from gallery images, followed by detection (FC cls/reg) and identification modules.
Text-based Person Search utilizes natural language descriptions. These systems must bridge the cross-modal gap, often employing Recurrent Neural Networks (RNNs) or Transformers to align textual attributes with visual features.
Key Challenges and Solutions
1. Deep Feature Representation Learning
To mitigate background clutter and occlusions, research has shifted from global feature extraction to local or mask-guided methods. For instance, APNet focuses on body parts, while SMG employs segmentation masks to isolate the subject from environmental noise.
2. Deep Metric Learning and the Query-Person Gap
The "query-person gap" refers to the difficulty of matching queries to gallery candidates across different views or modalities. A significant advancement is Online Instance Matching (OIM), which uses a lookup table to efficiently manage a large number of identities during training, enhancing the model's ability to distinguish between similar individuals.
3. Detection-Identification Inconsistency
A "perfect" bounding box for a detector (high IoU) may not be optimal for identification. End-to-end models like TCTS and NPSM address this by incorporating identity information directly into the detection process, allowing the identification task to guide the localization of the person.
Benchmarks and Performance
The field relies on several large-scale datasets for evaluation:
- Image-based: CUHK-SYSU (18k images) and PRW (11k frames). TCTS currently leads on CUHK-SYSU with a 93.9% mAP, while DKD excels on the PRW dataset (54.16% mAP).
- Text-based: CUHK-PEDES (40k images). ViTAA and TIMAM are top performers, with ViTAA achieving 55.97% Top-1 accuracy through fine-grained attribute alignment.
Future Directions
Despite progress, several frontiers remain:
- Multi-modal Search: Integrating both image and text queries for more precise retrieval.
- Zero-shot Search: Improving robustness when encountering unseen identities or attributes.
- Efficiency: Developing real-time search capabilities across massive image galleries without sacrificing accuracy.
Summary updated