Gallery Filter Network for Person Search

Lucas Jaffe, Avideh ZakhorWACV 2023

TL;DR

This paper introduces the Gallery Filter Network (GFN), a novel module that significantly improves person search efficiency and accuracy by first filtering out irrelevant gallery scenes before performing expensive person detection and re-identification. The GFN achieves state-of-the-art results on benchmark datasets and offers a more modular and efficient approach to person search.

What Problem Does This Paper Address?

Person search aims to find instances of a specific person within a large collection of gallery images or video frames. This is typically a two-step process: first, detecting all potential person bounding boxes in each gallery scene, and second, re-identifying the query person among these detected individuals. The computational cost of this process scales with the number of gallery scenes, making it crucial to reduce the search space. Existing methods often focus on improving detection or re-identification accuracy, but this paper tackles the efficiency bottleneck by proposing a way to quickly discard irrelevant gallery scenes before the computationally intensive detection and re-identification steps are even performed.

Methods

The core contribution is the Gallery Filter Network (GFN), a module designed to predict the likelihood of a query person appearing in a gallery scene. The GFN operates in two phases:

  1. Scene Retrieval (GFN Phase):
    • Query-Scene Embeddings: The GFN extracts embeddings for both the query person and each gallery scene.
  2. Similarity Scoring: It computes a similarity score between the query embedding and each scene embedding.
  3. Hard Thresholding: Gallery scenes with scores below a certain threshold are discarded, significantly reducing the search space. This step avoids the need for person detection in these filtered-out scenes.
  4. Person Retrieval (Standard Phase):
    • Detection: Person detection is performed only on the remaining, high-scoring gallery scenes.
  5. Re-identification: Person embeddings are extracted from detected bounding boxes and compared to the query.
  6. Score Weighting: The GFN also provides a scene score that can be used to re-weight the final scores of detected persons, further improving ranking accuracy.

The paper also introduces SeqNeXt, an improved version of the SeqNet person search model. SeqNeXt simplifies the detection pipeline and uses a ConvNeXt backbone for better performance and efficiency compared to the original ResNet50.

Datasets

The authors evaluate their methods on two standard person search benchmarks:

  • CUHK-SYSU: A diverse dataset containing images from handheld cameras and movies, with significant visual diversity.
  • PRW: A dataset consisting of video frames from surveillance cameras, focusing on cross-camera scenarios.

For more robust analysis, they also create train/validation splits for these datasets.

Key Insights and Ideas

  • Early Filtering is Key: The most significant insight is that filtering out irrelevant scenes early in the pipeline offers substantial computational savings, as detection is the most time-consuming step.
  • GFN as a Modular Component: The GFN is designed to be modular, meaning it can be integrated with various existing person search architectures, not just SeqNeXt.
  • Global Context Matters: The GFN leverages global scene context to inform the search, which is crucial for improving retrieval accuracy, especially in challenging scenarios.
  • Score Weighting Enhances Ranking: Beyond just filtering, the GFN's scene scores can be used to re-weight detected person scores, leading to better final rankings.
  • Robustness Across Conditions: The GFN demonstrates effectiveness across various challenging conditions, including occluded persons, low-resolution images, and cross-camera retrieval.

Key Results

  • State-of-the-Art Performance: The SeqNeXt+GFN combination achieves state-of-the-art results on both PRW and CUHK-SYSU datasets, improving mAP and top-1 accuracy. For instance, it achieved a +2.7% top-1 accuracy gain on the PRW dataset over previous best models.
  • Significant Efficiency Gains: The GFN provides substantial computational savings, with over 50% per-query cost savings on CUHK-SYSU compared to the same model without the GFN. This is attributed to avoiding detection in filtered scenes.
  • Improved Gallery Size Scalability: The GFN's benefits become more pronounced as the gallery size increases, as shown in Figure 4.
  • Robustness to Query Quality: The GFN performs well even with occluded or low-resolution query person views, as demonstrated in Table 1.

Limitations and Future Work

  • GFN Threshold Sensitivity: The effectiveness of hard thresholding depends on the discriminative power of GFN scores. For datasets with highly similar scenes (like PRW), the filtering benefit is less pronounced compared to datasets with more diverse scenes (like CUHK-SYSU).
  • Generalizability of GFN: While the GFN shows good cross-camera performance, further exploration into its generalization capabilities across even more diverse and challenging environments could be beneficial.
  • Societal Impact: The authors acknowledge the potential negative societal impact of person search technologies, particularly in surveillance applications. They advocate for open research to enable beneficial uses (e.g., finding missing persons) and to foster understanding for counteracting misuse.

The paper suggests that the GFN can be easily combined with other person search models, implying future work could explore its integration with a wider range of architectures.

Summary updated

BibTeX
@inproceedings{jaffe2023gallery,
  author = {Lucas Jaffe and Avideh Zakhor},
  title = {{Gallery Filter Network for Person Search}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2023},
}

Loading related content...

Figure