Diverse Knowledge Distillation for End-to-End Person Search

Xinyu Zhang, Xinlong Wang, Jia-Wang Bian, Chunhua Shen, Mingyu You

Introduction

Person search is a challenging computer vision task that involves both detecting individuals in a scene and identifying them across different camera views. Traditionally, this has been approached in two ways: two-step methods, which use separate models for detection and re-identification (Re-ID), and end-to-end methods, which integrate both tasks into a single joint network. While two-step methods are highly accurate, they are computationally expensive. Conversely, end-to-end models offer superior efficiency but have historically lagged in accuracy due to the inherent conflict between detection (finding common features) and Re-ID (finding unique features).

Comparison between standard end-to-end person search and the proposed Diverse Knowledge Distillation (DKD) framework.

The authors identify the Re-ID sub-network as the primary bottleneck in joint models. They demonstrate that detection performance in joint models is actually comparable to two-step systems; the performance gap is almost entirely due to the Re-ID head being weakened by task conflict. To address this, they propose Diverse Knowledge Distillation (DKD) to bridge the gap, leveraging a high-performance standalone Re-ID "teacher" to guide the joint "student" model during training.

Methodology

The proposed framework strengthens the Re-ID head of a joint model through two primary pillars: multi-faceted distillation and spatial-invariant augmentation.

Detailed architecture of the DKD framework, illustrating the joint training process with a pre-trained Re-ID teacher model and various distillation losses.

1. Diverse Knowledge Distillation (DKD)

Rather than simply mimicking final outputs, the student model learns from a pre-trained standalone Re-ID teacher through three distinct distillation paths:

  • Probability-aware KD: The student aligns its predicted class probabilities with those of the teacher.
  • Pair-wise relation-aware KD: The student learns to replicate the teacher’s similarity matrix, capturing how different individuals in a batch relate to one another.
  • Triplet-wise relation-aware KD: The student mimics the relative distance rankings between samples, ensuring that features for the same identity are clustered more closely than those of different identities.

2. Spatial-Invariant Augmentation

To address the sensitivity of joint models to misaligned bounding boxes produced by internal detectors, the authors introduce spatial-invariant techniques:

  • Image-level Spatial-invariant Augmentation (ISA): The teacher model is pre-trained on jittered bounding boxes to develop robustness against localization errors.
  • Feature-level Spatial-invariant Augmentation (FSA): During joint training, the RoIAlign crops are manually shifted to simulate detection inaccuracies, forcing the Re-ID head to produce consistent, robust features regardless of minor box misalignments.

Experimental Results

The DKD framework was evaluated on the CUHK-SYSU and PRW datasets, two standard benchmarks for person search in the wild.

  • State-of-the-Art Accuracy: On CUHK-SYSU, the model achieved a mean Average Precision (mAP) of 93.09%, matching the performance of top-tier two-step methods while maintaining the speed of an end-to-end system.
  • Real-time Efficiency: The system processes images at 124ms per frame, significantly faster than the 175ms required by comparable two-step pipelines.
  • Robust Scalability: As the search gallery size increases, the DKD method demonstrates superior stability compared to existing models, indicating that it learns more discriminative and robust identity features.

Conclusion and Future Work

The success of DKD highlights that the performance gap in end-to-end person search is not necessarily an architectural limitation but a supervision challenge. By using a teacher model only during training, the system gains the benefits of a complex two-step pipeline without any extra computational cost during inference.

While the method relies on a high-quality teacher and involves a more complex training process, it sets a new standard for efficient person search. Future directions may include exploring even more diverse forms of knowledge, such as structural or temporal information, to further bridge the gap to human-level search performance.

Summary updated

BibTeX
@inproceedings{zhang2021diverse,
  author = {Xinyu Zhang and Xinlong Wang and Jia-Wang Bian and Chunhua Shen and Mingyu You},
  title = {{Diverse Knowledge Distillation for End-to-End Person Search}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.1609/aaai.v35i4.16454},
}
Figure