Multigranular Visual-Semantic Embedding for Cloth-Changing Person Re-identification

Zan Gao, Hongwei Wei, Weili Guan, Weizhi Nie, Meng Liu, Meng Wang

TL;DR

The MVSE (Multigranular Visual-Semantic Embedding) framework tackles the "long-term" person re-identification (ReID) problem where subjects change their clothes between camera views. By combining multi-scale feature extraction with a "cloth desensitization" network and semantic part alignment, the model learns to identify people based on invariant body attributes rather than transient clothing colors.

What Problem Does This Paper Address?

Traditional Person Re-identification (ReID) systems operate under a "short-term" assumption: a person captured by Camera A will look virtually identical when captured by Camera B moments later. Consequently, these models rely heavily on the color and texture of clothing as primary identifiers.

In real-world surveillance, however, this assumption fails. Over longer time spans, people change jackets, take off hats, or switch bags. Standard ReID models typically collapse in these scenarios because they perceive a change in outfit as a change in identity. The authors address the challenge of Cloth-Changing Person Re-identification, aiming to extract "cloth-invariant" features—like body shape, gait-related proportions, and high-level semantic attributes—that remain consistent even when the wardrobe does not.

Methods

The authors propose a unified framework called MVSE, which consists of three primary modules built on top of a DenseNet121 backbone:

  1. Multigranular Representation (MGR): Inspired by the Multiple Granularity Network (MGN), this module splits feature maps into various horizontal stripes. This allows the model to learn both global body features and fine-grained local features (e.g., the head, torso, and legs) simultaneously, ensuring that if one part of the body is occluded or changed, other parts can still provide identity clues.
  2. Cloth Desensitization Network (CDN): This is the core innovation for handling outfit changes. It uses parallel convolutional layers to extract high-level human attributes. Crucially, it employs a "modulo operation" designed to detect the existence of certain physical attributes while shielding the network from the specific visual "noise" of different fabrics and colors.
  3. Partially Semantically Aligned Network (PSA): To handle pose changes and misalignments, the PSA module uses semantic segmentation (guided by HRNet pseudo-labels) to align visual features with specific body parts. This ensures the model compares "arm to arm" and "leg to leg" regardless of the subject's orientation or camera angle.

The model is trained using a combination of Classification Loss ($L_{id}$), Triplet Loss ($L_{tri}$), and a Partial Semantic Alignment Loss ($L_{part}$).

Datasets

The researchers evaluated MVSE on four specialized cloth-changing benchmarks:

  • LTCC (Long-Term Cloth-Changing): 152 identities with 478 different outfits captured over two months.
  • PRCC (Person Re-identification under Clothing Change): 221 identities across three camera views, specifically including "change" and "same" clothing subsets.
  • Celeb-reID: A large-scale dataset of 1,052 celebrities with over 34,000 images, where individuals naturally wear a high variety of clothes.
  • NKUP: 107 identities captured in an open university environment with significant variations in viewpoint and lighting.

Key Insights and Ideas

  • Visual-Semantic Synergy: The "aha moment" here is that visual features alone aren't enough when the "visuals" (clothes) are deceptive. By embedding semantic human attributes into the feature space, the model gains a "conceptual" understanding of the person that transcends their current outfit.
  • Desensitization vs. Feature Extraction: Rather than just trying to find better features, the CDN module explicitly tries to ignore (desensitize) the features that change (clothing) while amplifying those that don't (body structure).
  • Granularity as a Safety Net: By looking at the body at multiple scales (global, halves, and thirds), the model can find "islands of stability"—parts of the person that might not have changed, such as shoes or hair, even if the shirt is different.

Key Results

MVSE significantly outperformed both the baseline (DenseNet121) and previous state-of-the-art (SOTA) methods across all datasets:

  • LTCC: Achieved a 70.5% Rank-1 accuracy, a massive jump from the baseline's 27.2%.
  • PRCC: Reached 47.4% Rank-1 accuracy, outperforming the previous SOTA (SPT+ASE) by 13%.
  • Celeb-reID: Improved Rank-1 accuracy to 64.5% (a 1.5% improvement over ReIDCaps).
  • Ablation Success: The authors proved that the combination of MGR, CDN, and PSA is greater than the sum of its parts; removing any single module led to a measurable drop in performance.

Limitations and Future Work

While MVSE shows a dramatic improvement, the authors note that "shielding" the effects of clothing is an ongoing challenge. The current model relies on high-level attributes, but extremely bulky clothing (like heavy winter coats) can still mask body shape.

Future directions suggested by the paper include:

  • Developing even more robust "shielding" mechanisms to further isolate clothing noise.
  • Designing more sophisticated feature fusion modules to better integrate the multigranular data.
  • Exploring how these techniques might apply to video-based ReID where temporal gait information could provide additional cloth-invariant clues.

Summary updated

Abstract

To date, only a few works have focused on the cloth-changing person Re-identification (ReID) task, but since it is very difficult to extract generalized and robust features for representing people with different clothes, thus, their performances need to be improved. Moreover, visual-semantic information is also often ignored. To solve these issues, in this work, a novel multigranular visual-semantic embedding algorithm (MVSE) is proposed for cloth-changing person ReID, where visual semantic information and human attributes are embedded into the network, and the generalized features of human appearance can be well learned to effectively solve the problem of cloth-changing. Specifically, to fully represent a person with clothing changes, a multigranular feature representation scheme (MGR) is employed to adaptively extract multilevel and multigranular feature information, and then a cloth desensitization network (CDN) is designed to improve the feature robustness for the person with different clothes, where different high-level human attributes are fully utilized. Moreover, to further solve the issue of pose changes and occlusion under different camera perspectives, a partially semantically aligned network (PSA) is proposed to obtain the visual-semantic information that is used to align the human attributes. Most importantly, these three modules are jointly explored in a unified framework. Extensive experimental results on four cloth-changing person ReID datasets demonstrate that the MVSE algorithm can extract highly robust feature representations of cloth-changing persons, and it can outperform state-of-the-art cloth-changing person ReID approaches.
BibTeX
@inproceedings{gao2022multigranular,
  author = {Zan Gao and Hongwei Wei and Weili Guan and Weizhi Nie and Meng Liu and Meng Wang},
  title = {{Multigranular Visual-Semantic Embedding for Cloth-Changing Person Re-identification}},
  booktitle = {ACM Multimedia},
  year = {2022},
  doi = {10.1145/3503161.3547884},
}
Figure