Joint Person Objectness and Repulsion for Person Search

Hantao Yao, Changsheng Xu

Abstract

Person search aims to locate a specific individual within unconstrained scene images, combining the tasks of person detection and person matching. Traditional Detection-Matching frameworks often struggle with "distractor" images—non-person background regions or similar-looking individuals. This paper introduces the OR Similarity metric, which integrates Objectness and Repulsion information to refine visual matching. The objectness term leverages detector confidence to penalize false detections, while the repulsion term uses the probe's spatial neighbors as "hard negatives" to prevent identity confusion. Evaluated on the CUHK-SYSU and PRW datasets, the proposed method consistently improves search accuracy across various description models, raising mAP from 92.32% to 93.23% on CUHK-SYSU and from 50.91% to 52.30% on PRW.

Introduction

The task of person search involves finding a specific individual (the "probe") within a large gallery of scene images. This is typically approached as a two-stage process: first, a detector identifies all person-like regions in the gallery images; second, a matching model compares these regions to the probe image to find the best match. However, this framework faces significant challenges from "distractors"—regions that the detector mistakenly identifies as people or individuals who look similar to the target but are not the same person.

A probe image with the target person identified by a bounding box.

The Challenges: Distractors and Identity Confusion

Current systems often fail because they treat every detection as a valid person and ignore the spatial context of the probe.

  1. Objectness Issues: Detectors often produce high-confidence boxes for background clutter or partial body parts. If these "distractor" boxes happen to have high visual similarity to the probe, they result in false positives.
  2. Identity Confusion (Repulsion): In crowded scenes, a probe person is often surrounded by others. These neighbors are guaranteed to have different IDs. Without accounting for this, a system might match the probe to a gallery image that actually represents one of the probe's neighbors.
Visualization of the person search task, matching a probe against a gallery of scene images. The presence of neighbors in the probe image provides critical context for the Repulsion term.

Proposed Method: OR Similarity

To address these issues, the researchers propose OR Similarity, a scoring modification that can be applied to existing person search models without retraining. The final similarity score $S_{or}$ is the product of three terms:

  • Visual Similarity ($S$): The standard similarity between the feature embeddings of the probe and the gallery candidate.
  • Objectness Term ($S^o$): This term incorporates the confidence score from the person detector. By applying an exponential function to the detection score, the system heavily penalizes candidates that the detector is less certain are actual people, effectively filtering out background noise.
  • Repulsion Term ($S^r$): This term leverages the fact that the probe person and their neighbors in the original frame must have different identities. If a gallery candidate is found to be more similar to one of the probe's neighbors than to the probe itself, the repulsion term drastically reduces its similarity score, preventing identity confusion.

Key Results

The OR Similarity metric was evaluated using six different description models on two major benchmarks. The addition of OR Similarity consistently improved performance across the board:

  • CUHK-SYSU: Using the MGN model, the mean Average Precision (mAP) rose from 92.32% to 93.23%.
  • PRW: On the more difficult PRW dataset, mAP increased from 50.91% to 52.30%.

Qualitative analysis demonstrated that the repulsion term is particularly effective in cases of occlusion. When a probe person is partially hidden by a neighbor, standard models often latch onto the neighbor's features; OR Similarity successfully "repels" these incorrect matches.

Conclusion

By explicitly modeling person objectness and spatial repulsion, the proposed OR Similarity metric provides a robust way to prune distractors in person search. Its "plug-and-play" nature allows it to enhance a wide variety of existing detection-matching frameworks. Future work may involve integrating these constraints directly into the representation learning phase to create even more discriminative person embeddings.

Summary updated

Abstract

Person search targets to search the probe person from the unconstrainted scene images, which can be treated as the combination of person detection and person matching. However, the existing methods based on the Detection-Matching framework ignore the person objectness and repulsion (OR) which are both beneficial to reduce the effect of distractor images. In this paper, we propose an OR similarity by jointly considering the objectness and repulsion information. Besides the traditional visual similarity term, the OR similarity also contains an objectness term and a repulsion term. The objectness term can reduce the similarity of distractor images that not contain a person and boost the performance of person search by improving the ranking of positive samples. Because the probe person has a different person ID with its \emph{neighbors}, the gallery images having a higher similarity with the \emph{neighbors of probe} should have a lower similarity with the probe person. Based on this repulsion constraint, the repulsion term is proposed to reduce the similarity of distractor images that are not most similar to the probe person. Treating the Faster R-CNN as the person detector, the OR similarity is evaluated on PRW and CUHK-SYSU datasets by the Detection-Matching framework with six description models. The extensive experiments demonstrate that the proposed OR similarity can effectively reduce the similarity of distractor samples and further boost the performance of person search, e.g., improve the mAP from 92.32% to 93.23% for CUHK-SYSY dataset, and from 50.91% to 52.30% for PRW datasets.
BibTeX
@article{yao2021joint,
  author = {Hantao Yao and Changsheng Xu},
  title = {{Joint Person Objectness and Repulsion for Person Search}},
  journal = {IEEE Trans. Image Process.},
  year = {2021},
  url = {https://openreview.net/forum?id=Akz6jqEzeM9},
}
Figure