Abstract
Person search aims to locate and identify a specific person from uncropped gallery images. This task faces significant challenges from background clutter and body part misalignment. This paper proposes BPNet, an end-to-end framework that integrates person detection and re-identification (ReID) with auxiliary tasks like instance segmentation and keypoint detection to achieve robust "thorough body perception."
The Challenges of Person Search
Person search is significantly more complex than standard person re-identification (ReID). While standard ReID provides neatly cropped images for matching, person search requires finding a "probe" person within raw, uncropped gallery images. This introduces two major hurdles:
- Background Clutter: Detectors often include environmental elements (trees, cars, other people) inside the bounding box, which confuses the ReID feature extractor.
- Misalignment: Due to detector errors or dramatic changes in human pose, different body parts may be incorrectly compared (e.g., matching a leg to a patch of pavement), leading to false matches.
Methods: The BPNet Architecture
The authors propose the Body Perception Network (BPNet), a multi-task architecture built on a ResNet-50 and Feature Pyramid Network (FPN) backbone. As shown in the architecture diagram, it features four parallel branches: RCNN (detection), ReID, Mask (segmentation), and Kp (keypoints).
Two novel components drive its performance:
- Foreground Feature Enhance Module (FFEM): This module utilizes the segmentation branch to create a foreground mask. By applying this mask to the ReID feature maps via a Hadamard product, the network learns to suppress background noise and focus purely on the person's appearance.
- Keypoints-Guided Learning Algorithm: Instead of using rigid grids to divide a person into parts, BPNet uses detected keypoints (shoulders, hips, etc.) to define "abstract regions" like the upper body and torso. This ensures that the model compares corresponding body parts even when a person is crouching or viewed from unusual angles.
To ensure practical efficiency, the system implements a "light-head" design, replacing heavy fully-connected layers with cascade convolutions, making the model twice as fast as previous baselines.
Experimental Results
The model was evaluated on two gold-standard benchmarks:
- CUHK-SYSU: A large-scale dataset with over 18,000 images and 8,000+ identities. BPNet reached 90.5% Rank-1 accuracy and 88.4% mAP, outperforming previous state-of-the-art methods like QEEPS.
- PRW (Person Search in the Wild): A challenging dataset captured from six synchronized cameras. BPNet achieved 87.9% Rank-1 accuracy, a significant jump over the previous best of 76.7%.
Beyond accuracy, the "light-head" baseline proved to be roughly 2x faster than original "heavy-head" models while maintaining superior performance.
Key Insights
- Auxiliary Tasks as Regularizers: Adding segmentation and keypoint detection—tasks not strictly required for the final output—makes ReID features much more robust. It forces the network to "understand" human anatomy rather than just memorizing pixel patterns.
- Dynamic Part Alignment: By using keypoints to define body regions, the model becomes invariant to the cropping errors that usually plague detection-based systems. If the detector cuts off the feet, the keypoint-guided logic knows not to attempt a match in that empty space.
Limitations and Future Work
The authors currently rely on pseudo-labels generated from COCO-trained models because the target datasets lack manual segmentation and keypoint annotations. There is a "domain shift" between COCO (general objects) and person search datasets (surveillance footage). Future work involving manual annotations specifically for person search datasets could further boost performance. Additionally, the multi-task approach increases training complexity, requiring careful balancing of five different loss weights.
Summary updated