SAT: Scale-Augmented Transformer for Person Search

Mustansar Fiaz, Hisham Cholakkal, Rao Muhammad Anwer, Fahad Shahbaz Khan

Introduction

Person search is a complex computer vision task that combines pedestrian detection and person re-identification (ReID) into a single pipeline. Unlike standard ReID, which starts with pre-cropped images, person search requires models to find a target individual within large, unconstrained gallery images. The Scale-Augmented Transformer (SAT) introduces a three-stage cascaded framework designed to handle the inherent conflicts and scale variations in this domain.

What Problem Does This Paper Address?

The authors identify three primary challenges in existing person search methods:

  1. Task Conflict: Detection aims for inter-class commonality (distinguishing people from background), while ReID requires intra-class discrimination (distinguishing between different people).
  2. Scale and Pose Variance: Pedestrians appear at varying distances and in diverse orientations, often suffering from partial occlusion.
  3. Feature Sub-optimality: Shared parameters in "one-step" models often lead to a performance compromise between detection and ReID.
Visualizing the person search task: detecting and re-identifying individuals across diverse scenes.

The SAT Framework

To overcome these hurdles, the SAT framework adopts a coarse-to-fine refinement strategy built on three core innovations:

Decoupled Norm-Aware Embedding (NAE)

SAT utilizes separate embeddings for detection and ReID. By employing Norm-Aware Embedding, the model uses the feature magnitude (norm) for detection confidence and the feature direction (angle) for identity representation. This decoupling prevents the two tasks from interfering with each other's learned parameters.

Scale-Augmented Transformer (SAT) Block

The SAT block is a hybrid module that combines the local inductive bias of CNNs with the global context of Transformers. It features a Convolutional Encoder followed by a Scale Modulator. This modulator uses depth-wise convolutions with multiple kernel sizes (1x1, 3x3, 5x5) to capture multi-scale features within the Region of Interest (RoI), ensuring robust performance regardless of the person's distance from the camera.

The SAT framework's coarse-to-fine refinement process for a single pedestrian.

Augmented Mixing for Occlusion

To improve resilience against occlusions, the authors introduce "Augmented Mixing." This technique synthetically creates misalignments by shifting feature tokens during training. This forces the model to learn identity representations that do not rely on a perfect, unobstructed view of the entire body.

Key Results and Performance

The SAT framework achieves state-of-the-art results on major benchmarks:

  • PRW Dataset: SAT reached 54.5% mAP, marking a significant 5.0% absolute improvement over the previous leader, PSTR.
  • CUHK-SYSU: The model achieved 94.4% mAP and 94.8% Top-1 accuracy, outperforming both one-step and two-step competitors.
  • Efficiency: Despite its multi-stage design, SAT maintains high efficiency, processing images at 105ms on a V100 GPU—nearly twice as fast as some older real-time methods like MGTS.

Conclusion and Future Directions

The Scale-Augmented Transformer demonstrates that a cascaded, scale-aware approach can effectively resolve the tension between detection and re-identification. While the model sets new performance standards, failure case analysis indicates that extreme occlusion remains a challenge. Future work may explore the integration of temporal data for video-based search and further latency optimizations for the cascaded architecture.

Summary updated

Abstract

Person search is a challenging computer vision problem where the objective is to simultaneously detect and reidentify a target person from the gallery of whole scene images captured from multiple cameras. Here, the challenges related to underlying detection and re-identification tasks need to be addressed along with a joint optimization of these two tasks. In this paper, we propose a three-stage cascaded Scale-Augmented Transformer (SAT) person search framework. In the three-stage design of our SAT framework, the first stage performs person detection whereas the last two stages performs both detection and re-identification. Considering the contradictory nature of detection and re-identification, in the last two stages, we introduce separate norm feature embeddings for the two tasks to reconcile the relationship between them in a joint person search model. Our SAT framework benefits from the attributes of convolutional neural networks and transformers by introducing a convolutional encoder and a scale modulator within each stage. Here, the convolutional encoder increases the generalization ability of the model whereas the scale modulator performs context aggregation at different granularity levels to aid in handling pose/scale variations within a region of interest. To further improve the performance during occlusion, we apply shifting augmentation operations at each granularity level within the scale modulator. Experimental results on challenging CUHK-SYSU [35] and PRW [47] datasets demonstrate the favorable performance of our method compared to state-of-the-art methods. Our source code and trained models are available at this https URL.
BibTeX
@inproceedings{fiaz2023sat,
  author = {Mustansar Fiaz and Hisham Cholakkal and Rao Muhammad Anwer and Fahad Shahbaz Khan},
  title = {{SAT: Scale-Augmented Transformer for Person Search}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2023},
  doi = {10.1109/wacv56688.2023.00480},
}
Figure