FastPR: One-stage Semantic Person Retrieval via Self-supervised Learning

Meng Sun, Ju Ren, Xin Wang, Wenwu Zhu, Yaoxue Zhang

Abstract

Semantic person retrieval aims to locate a specific person in an image with the query of semantic descriptions, which has shown great significance in surveillance and security applications. Prior arts commonly adopt a two-stage method that first extracts the persons with a pretrained detector and then finds the target matching the descriptions optimally. However, existing works suffer from high computational complexity and low recall rate caused by error accumulation in the two-stage inference. To solve the problems, we propose FastPR, a one-stage semantic person retrieval method via self-supervised learning, to optimize the person localization and semantic retrieval simultaneously. Specifically, we propose a dynamic visual-semantic alignment mechanism which utilizes grid-based attention to fuse the cross-modal features, and employs a label prediction proxy task to constrain the attention process. To tackle the challenges that real-world surveillance images may suffer from low-resolution and occlusion, and the target persons may be within a crowd, we further propose a dual-granularity person localization module through designing an upsampling reconstruction proxy task to enhance the local feature of the target person in the fused features, followed by a tailored offset prediction proxy task to make the localization network capable of accurately identifying and distinguishing the target person in a crowd. Experimental results demonstrate that FastPR achieves the best retrieval accuracy compared to the state-of-the-art baseline methods, with over 15 times inference time improvement.
BibTeX
@inproceedings{sun2022fastpr,
  author = {Meng Sun and Ju Ren and Xin Wang and Wenwu Zhu and Yaoxue Zhang},
  title = {{FastPR: One-stage Semantic Person Retrieval via Self-supervised Learning}},
  booktitle = {ACM Multimedia},
  year = {2022},
  doi = {10.1145/3503161.3548274},
}

Loading related content...

Figure