Swap Path Network for Robust Person Search Pre-Training

Lucas Jaffe, Avideh ZakhorWACV 2025

TL;DR

This paper introduces Swap Path Net (SPNet), a novel framework for pre-training person search models. SPNet leverages a "query-centric" pre-training approach that is more robust to noisy labels and can be trained with weakly-supervised data, leading to state-of-the-art results on standard person search benchmarks.

What Problem Does This Paper Address?

Person search, the task of finding a specific person within a collection of images or videos, typically involves two main components: detecting all people and then re-identifying them. While pre-training deep learning backbones for image classification is a well-established practice, pre-training the entire person search pipeline, including detection and re-identification modules, has been largely unexplored. Existing methods often rely on standard ImageNet pre-training for the backbone, leaving other crucial parts of the model randomly initialized. Furthermore, real-world datasets often suffer from label noise (e.g., missing or inaccurate bounding boxes), which can negatively impact model performance, especially during pre-training. This paper aims to bridge this gap by proposing an end-to-end pre-training framework that is robust to label noise and improves overall person search performance.

Methods

The core contribution is the Swap Path Network (SPNet), a model designed for both query-centric (QC) and object-centric (OC) operations. The key innovation lies in the query-centric pre-training strategy.

  • Query-Centric (QC) vs. Object-Centric (OC) Operation:
    • OC: Standard approach where a detector finds all objects, and then queries are matched against these detections.
  • QC: The model uses the query image directly during the detection process, generating proposals tailored to the query. This is computationally more expensive during inference but is found to be more robust for pre-training.
  • SPNet Architecture: SPNet consists of a backbone, a Feature Pyramid Network (FPN), a Box Embedder, and an SP Block. The SP Block is where the QC and OC pathways diverge.
  • QC Pathway: Takes a query embedding and anchor embeddings to predict localization offsets and class logits. This pathway is robust to label noise because it focuses on matching a specific query.
  • OC Pathway: Used for fine-tuning and inference. It predicts a pseudo-query embedding from anchor embeddings, allowing for efficient object-centric operation.
  • Pre-training Strategy: The paper advocates for pre-training the entire SPNet model using the QC pathway on a large, weakly-labeled dataset (COCOPersons). This allows the model to learn robust features that generalize well to the person search task.
  • Weakly-Supervised Pre-training: The QC pre-training can be performed using only bounding box annotations, without requiring explicit identity labels across images. This is achieved by using data augmentation to create correspondences between different views of the same scene.

Datasets

  • Pre-training: COCOPersons dataset (64k scenes, 257k person box annotations). This dataset is used for weakly-supervised pre-training.
  • Fine-tuning/Evaluation:
    • CUHK-SYSU dataset
    • PRW dataset

Key Insights and Ideas

  • Query-Centric Pre-training is Key: The most significant insight is that pre-training using a query-centric approach, rather than the standard object-centric one, leads to substantial improvements in person search performance. This is attributed to its inherent robustness to label noise.
  • Weak Supervision is Sufficient for Pre-training: The framework demonstrates that effective pre-training can be achieved using only bounding box annotations, without the need for expensive identity-level labels, making it more scalable.
  • End-to-End Pre-training Benefits All Modules: Unlike prior work that only pre-trained backbones, SPNet pre-trains the entire model (backbone, detection, and re-id heads), leading to better feature learning across the board.
  • QC for Pre-training, OC for Fine-tuning: The strategy of using the robust QC pathway for pre-training and then switching to the efficient OC pathway for fine-tuning and inference offers the best of both worlds.
  • Robustness to Label Noise: The QC pre-training approach shows a clear advantage when datasets contain missing or spurious annotations, a common issue in real-world scenarios.

Key Results

  • State-of-the-Art Performance: SPNet-L, with QC pre-training on COCOPersons, achieves state-of-the-art results on both CUHK-SYSU (96.4% mAP) and PRW (61.2% mAP) datasets.
  • Efficiency: The proposed pre-training method is shown to be more efficient than previous backbone-only pre-training alternatives, achieving comparable or better results in less training time.
  • Robustness: QC pre-training significantly outperforms OC pre-training when tested with noisy labels, demonstrating its practical advantage.
  • Speed: SPNet-L with a ResNet50 backbone achieves competitive inference speeds (27.6 FPS), outperforming other models like SeqNeXt.

Limitations and Future Work

  • Computational Cost of QC Inference: While QC pre-training is beneficial, the QC pathway itself is computationally more expensive for inference compared to OC. The paper addresses this by using OC for fine-tuning and inference.
  • Dataset Size for Pre-training: The paper notes that while COCOPersons is used for pre-training, larger datasets might yield further improvements.
  • Future Work: The authors suggest exploring further variations of the QC pre-training objective and investigating its applicability to other person-centric tasks. They also highlight the potential for further optimization of the SPNet architecture.

Summary updated

BibTeX
@inproceedings{jaffe2025swap,
  author = {Lucas Jaffe and Avideh Zakhor},
  title = {{Swap Path Network for Robust Person Search Pre-Training}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2025},
}

Loading related content...

Figure