End-to-End Thorough Body Perception for Person Search

Kun Tian, Houjing Huang, Yun Ye, Shiyu Li, Jinbin Lin, Guan HuangAAAI 2020 · DOI

Abstract

Person search aims to locate and identify a specific person from uncropped gallery images. This task faces significant challenges from background clutter and body part misalignment. This paper proposes BPNet, an end-to-end framework that integrates person detection and re-identification (ReID) with auxiliary tasks like instance segmentation and keypoint detection to achieve robust "thorough body perception."

Person search is significantly more complex than standard person re-identification (ReID). While standard ReID provides neatly cropped images for matching, person search requires finding a "probe" person within raw, uncropped gallery images. This introduces two major hurdles:

  1. Background Clutter: Detectors often include environmental elements (trees, cars, other people) inside the bounding box, which confuses the ReID feature extractor.
  2. Misalignment: Due to detector errors or dramatic changes in human pose, different body parts may be incorrectly compared (e.g., matching a leg to a patch of pavement), leading to false matches.
Common challenges in person search: background noise, detection errors, and pose variations leading to matching errors.

Methods: The BPNet Architecture

The authors propose the Body Perception Network (BPNet), a multi-task architecture built on a ResNet-50 and Feature Pyramid Network (FPN) backbone. As shown in the architecture diagram, it features four parallel branches: RCNN (detection), ReID, Mask (segmentation), and Kp (keypoints).

The BPNet architecture featuring shared backbone and four specialized branches: RPN/RCNN, ReID, Mask, and Keypoint (Kp).

Two novel components drive its performance:

  • Foreground Feature Enhance Module (FFEM): This module utilizes the segmentation branch to create a foreground mask. By applying this mask to the ReID feature maps via a Hadamard product, the network learns to suppress background noise and focus purely on the person's appearance.
  • Keypoints-Guided Learning Algorithm: Instead of using rigid grids to divide a person into parts, BPNet uses detected keypoints (shoulders, hips, etc.) to define "abstract regions" like the upper body and torso. This ensures that the model compares corresponding body parts even when a person is crouching or viewed from unusual angles.

To ensure practical efficiency, the system implements a "light-head" design, replacing heavy fully-connected layers with cascade convolutions, making the model twice as fast as previous baselines.

Experimental Results

The model was evaluated on two gold-standard benchmarks:

  • CUHK-SYSU: A large-scale dataset with over 18,000 images and 8,000+ identities. BPNet reached 90.5% Rank-1 accuracy and 88.4% mAP, outperforming previous state-of-the-art methods like QEEPS.
  • PRW (Person Search in the Wild): A challenging dataset captured from six synchronized cameras. BPNet achieved 87.9% Rank-1 accuracy, a significant jump over the previous best of 76.7%.

Beyond accuracy, the "light-head" baseline proved to be roughly 2x faster than original "heavy-head" models while maintaining superior performance.

Key Insights

  • Auxiliary Tasks as Regularizers: Adding segmentation and keypoint detection—tasks not strictly required for the final output—makes ReID features much more robust. It forces the network to "understand" human anatomy rather than just memorizing pixel patterns.
  • Dynamic Part Alignment: By using keypoints to define body regions, the model becomes invariant to the cropping errors that usually plague detection-based systems. If the detector cuts off the feet, the keypoint-guided logic knows not to attempt a match in that empty space.

Limitations and Future Work

The authors currently rely on pseudo-labels generated from COCO-trained models because the target datasets lack manual segmentation and keypoint annotations. There is a "domain shift" between COCO (general objects) and person search datasets (surveillance footage). Future work involving manual annotations specifically for person search datasets could further boost performance. Additionally, the multi-task approach increases training complexity, requiring careful balancing of five different loss weights.

Summary updated

BibTeX
@inproceedings{tian2020end,
  author = {Kun Tian and Houjing Huang and Yun Ye and Shiyu Li and Jinbin Lin and Guan Huang},
  title = {{End-to-End Thorough Body Perception for Person Search}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.1609/aaai.v34i07.6886},
}
Figure