Vision Shared and Representation Isolated Network for Person Search

Yang Liu, Yingping Li, Chengyu Kong, Yuqiu Kong, Shenglan Liu, Feilong WangIJCAI 2022 · DOI

Introduction: The Representation Paradox

Person search is a complex computer vision task that requires simultaneously locating pedestrians in a scene (detection) and identifying specific individuals across different camera views (re-identification, or re-ID). While one-stage methods—which perform both tasks in a single pass—are computationally efficient, they often suffer from an "optimization conflict."

As illustrated in the figure below, detection is a coarse-grained task that requires the model to find similarities between all people to classify them as the same category. In contrast, re-ID is a fine-grained task requiring high discrimination to distinguish between individuals. When these tasks share the same feature representation, the resulting "representation paradox" limits the performance of both subtasks.

Conceptual illustration of the coarse-grained detection subtask versus the fine-grained re-ID subtask.

The VSRI Network Architecture

To resolve this conflict, the authors propose the Vision Shared and Representation Isolated (VSRI) network. The core philosophy is to decouple the conflicting requirements at the architectural level by using a shared "stem" for general visual features while isolating task-specific branches.

Isolated Branches and Feature Fusion

The VSRI network utilizes a shared backbone (typically ResNet-50) up to the conv4_x layer. Following this stem, the architecture splits into two independent paths:

  1. Detection Branch: Optimized for consistency to identify all pedestrian bounding boxes.
  2. Re-ID Branch: Optimized for discrimination to extract unique identity signatures.

Multi-Level Feature Fusion (MLFF)

A critical innovation in VSRI is the Multi-Level Feature Fusion (MLFF) module. Because re-ID relies on subtle details (e.g., clothing patterns) that are often lost in deeper semantic layers, the MLFF module pulls shallow features from earlier stages of the backbone and fuses them with deeper features.

Detailed architecture of the VSRI network, highlighting the Multi-Level Feature Fusion (MLFF) module and its sub-components: SPFF, MADE, and FDCI.

As shown in the detailed architecture above, the MLFF module consists of three primary components:

  • Spatial Pyramid Feature Fusion (SPFF): Aligns and concatenates features from multiple stages (C1-C4) of the backbone.
  • Multi-head Attention Driven Extraction (MADE): Employs self-attention to capture global relationships across the body, helping the model focus on the person while ignoring background clutter.
  • Feature Decomposition and Cascaded Integration (FDCI): Uses multi-scale convolutions to capture local details at various receptive fields, ensuring fine-grained identifiers are preserved.

Experimental Results

The VSRI network was evaluated on two major benchmarks, achieving state-of-the-art (SOTA) results for one-stage frameworks:

  • PRW (Person Search in the Wild): VSRI achieved 52.9% mAP, outperforming previous leaders like DKD (50.5%). Ablation studies showed that the MLFF module alone contributed an 11.6-point increase in mAP over the baseline.
  • CUHK-SYSU: The model reached 93.4% mAP and 94.1% Top-1 accuracy, demonstrating robust performance across diverse street scenes and movie stills.

Conclusion

The VSRI network successfully addresses the fundamental optimization conflict in person search by isolating task-specific representations. By leveraging shallow, fine-grained features through the MLFF module and decoupling the detection and re-ID heads, the framework achieves superior accuracy without sacrificing the efficiency of a one-stage approach. Future work may involve integrating more advanced detection backbones or optimizing the fusion modules for real-time edge deployment.

Summary updated

BibTeX
@inproceedings{liu2022vision,
  author = {Yang Liu and Yingping Li and Chengyu Kong and Yuqiu Kong and Shenglan Liu and Feilong Wang},
  title = {{Vision Shared and Representation Isolated Network for Person Search}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2022},
  doi = {10.24963/ijcai.2022/170},
}
Figure