Person Search Challenges and Solutions: A Survey

Xiangtan Lin, Pengzhen Ren, Yun Xiao, Xiaojun Chang, Alex Hauptmann

Abstract

Person search has drawn increasing attention due to its real-world applications and research significance. It aims to find a probe person in a gallery of scene images, with applications ranging from criminal investigations to missing person searches. This survey analyzes highly influential methods across three significant challenges: discriminative person features, the query-person gap, and detection-identification inconsistency. We summarize evaluation results and discuss promising future research directions.

Introduction: The Person Search Problem

In surveillance and forensics, the task of finding a specific individual is rarely as simple as matching two perfectly cropped photos. While standard Person Re-identification (Re-ID) assumes that detection is already completed and focuses on matching pre-cropped images, Person Search is an "end-to-end" task. It requires models to simultaneously detect people within raw scene images and identify the specific target.

This task is significantly harder due to several factors:

  1. Detection Errors: Poorly cropped bounding boxes (e.g., missing limbs or excessive background) can degrade identification performance.
  2. Conflicting Objectives: Detection aims for commonality (finding "any person"), whereas identification requires distinctiveness (finding "this specific person").
  3. The Modality Gap: In text-based search, models must bridge the semantic gap between natural language descriptions and raw pixel data.

Methodological Pillars

The survey categorizes existing literature into three strategic pillars designed to address these complexities. These are broadly divided into image-based and text-based query modalities.

Architectures for image-based and text-based person search pipelines.

1. Deep Feature Representation Learning

These methods focus on extracting "clutter-proof" features. While early approaches relied on global features, modern techniques utilize local/part-based features—dividing the body into regions—or context cues from the surrounding scene to enhance discriminability against complex backgrounds.

2. Deep Metric Learning

Metric learning addresses the "query-person gap." A landmark development in this area is the Online Instance Matching (OIM) loss. Unlike standard Softmax, which struggles with the vast number of identities in person search datasets, OIM maintains a lookup table of features to efficiently compare queries against both labeled and unlabeled identities.

3. Identity-driven Detection

This novel category integrates identity information directly into the detection process. Rather than treating detection and identification as isolated steps, these models use identity cues to "guide" the detector. For instance, Knowledge Distillation (KD) can be used where a pre-trained identification model teaches the detection network which regions of a bounding box are most critical for recognition.

Key Challenges and Insights

The survey highlights the Inconsistency Paradox: a high "Intersection over Union" (IoU) score—the standard metric for detection—does not always correlate with better identification. A box that perfectly bounds a person might include background noise that confuses the ID model.

Furthermore, while text-based search is highly practical (as it doesn't require an initial photo), it is technically demanding. Top-performing models like ViTAA succeed by decomposing both text and images into fine-grained attributes (e.g., "short hair," "blue jeans") to achieve precise cross-modal alignment.

Benchmarks and State-of-the-Art

The field is evaluated across several primary benchmarks:

  • CUHK-SYSU & PRW: The standard large-scale benchmarks for image-based search.
  • LSPM (Large-Scale Person Matching): Designed to test performance on partial body detections.
  • CUHK-PEDES: The gold standard for text-based search, featuring over 80,000 descriptions.

Currently, image-based methods like TCTS and DKD lead with mAP scores over 93% on CUHK-SYSU. In the text-based domain, ViTAA holds the top spot, demonstrating the superiority of attribute-level alignment over global matching.

Future Frontiers

The authors identify several open research directions:

  • Multi-modal Search: Combining image and text queries (e.g., "Find this person, but wearing a different hat").
  • Zero-Shot Search: Improving the ability to find identities not seen during training.
  • Efficiency: Optimizing models for real-time deployment across large-scale camera networks without sacrificing precision.

Summary updated

Abstract

Person search has drawn increasing attention due to its real-world applications and research significance. Person search aims to find a probe person in a gallery of scene images with a wide range of applications, such as criminals search, multicamera tracking, missing person search, etc. Early person search works focused on image-based person search, which uses person image as the search query. Text-based person search is another major person search category that uses free-form natural language as the search query. Person search is challenging, and corresponding solutions are diverse and complex. Therefore, systematic surveys on this topic are essential. This paper surveyed the recent works on image-based and text-based person search from the perspective of challenges and solutions. Specifically, we provide a brief analysis of highly influential person search methods considering the three significant challenges: the discriminative person features, the query-person gap, and the detection-identification inconsistency. We summarise and compare evaluation results. Finally, we discuss open issues and some promising future research directions.
BibTeX
@inproceedings{lin2021person,
  author = {Xiangtan Lin and Pengzhen Ren and Yun Xiao and Xiaojun Chang and Alex Hauptmann},
  title = {{Person Search Challenges and Solutions: A Survey}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2021},
  url = {https://openreview.net/forum?id=46TAj9Rzk1},
}
Figure