Multigranular Visual-Semantic Embedding for Cloth-Changing Person Re-identification

Zan Gao, Hongwei Wei, Weili Guan, Weizhi Nie, Meng Liu, Meng Wang

Abstract

Traditional person re-identification (ReID) systems often fail when subjects change their clothing between camera views, as they rely heavily on color and texture. We present the Multigranular Visual-Semantic Embedding (MVSE) framework, designed for "long-term" ReID. MVSE utilizes a combination of multi-scale feature extraction, a cloth desensitization network, and semantic part alignment to learn identity-defining attributes that remain invariant across wardrobe changes. Experimental results on four major benchmarks demonstrate that MVSE significantly outperforms existing state-of-the-art methods.

Introduction

Person Re-identification (ReID) is a critical task in surveillance and security, aiming to match individuals across non-overlapping camera views. Most existing models operate under a "short-term" assumption where the subject's appearance remains constant. However, in real-world scenarios spanning hours or days, individuals frequently change their attire, rendering color-based features unreliable.

Examples of cloth-changing scenarios for two individuals.

The challenge of Cloth-Changing Person Re-identification requires models to move beyond transient visual cues and focus on "cloth-invariant" features such as body shape, gait proportions, and high-level semantic attributes.

Methodology

The MVSE framework is built upon a DenseNet121 backbone and incorporates three specialized modules to capture robust identity features.

Multigranular Representation (MGR)

The MGR module, inspired by Multiple Granularity Networks, partitions feature maps into horizontal stripes of varying scales. By learning both global and fine-grained local features simultaneously, the model ensures that identity clues can be recovered even if specific body parts are occluded or modified by clothing.

Cloth Desensitization Network (CDN)

The CDN is the primary mechanism for handling outfit variations. It employs parallel convolutional layers to extract high-level human attributes. A unique "modulo operation" is used to detect the presence of physical attributes while shielding the network from the specific visual noise associated with different fabrics and colors.

Partially Semantically Aligned Network (PSA)

To address pose variations and spatial misalignments, the PSA module utilizes semantic segmentation (via HRNet pseudo-labels) to align visual features with specific body parts. This ensures that the model performs consistent comparisons (e.g., comparing legs to legs) regardless of the subject's orientation.

Experiments and Results

The MVSE framework was evaluated on four specialized cloth-changing datasets: LTCC, PRCC, Celeb-reID, and NKUP.

On the LTCC (Long-Term Cloth-Changing) dataset, MVSE achieved a Rank-1 accuracy of 70.5%, a substantial improvement over the 27.2% baseline. On the PRCC dataset, it reached 47.4% Rank-1 accuracy, surpassing the previous state-of-the-art by 13%. Similar gains were observed on Celeb-reID (64.5% Rank-1) and NKUP. Ablation studies confirmed that each module (MGR, CDN, and PSA) contributes significantly to the overall performance.

Conclusion

The MVSE framework represents a significant step forward in long-term person re-identification by effectively "desensitizing" models to clothing changes. By integrating visual and semantic information, the model achieves superior robustness. Future work will focus on handling extremely bulky clothing and exploring temporal gait information in video-based ReID to further enhance cloth-invariant identification.

Summary updated

BibTeX
@inproceedings{gao2022multigranular,
  author = {Zan Gao and Hongwei Wei and Weili Guan and Weizhi Nie and Meng Liu and Meng Wang},
  title = {{Multigranular Visual-Semantic Embedding for Cloth-Changing Person Re-identification}},
  booktitle = {ACM Multimedia},
  year = {2022},
  doi = {10.1145/3503161.3547884},
}
Figure