Fine-Grained Shape-Appearance Mutual Learning for Cloth-Changing Person Re-Identification

Peixian Hong, Tao Wu, Ancong Wu, Xintong Han, Wei-Shi ZhengCVPR 2021 · OpenReview

Abstract

Recently, person re-identification (Re-ID) has achieved great progress. However, current methods largely depend on color appearance, which is not reliable when a person changes the clothes. Cloth-changing Re-ID is challenging since pedestrian images with clothes change exhibit large intra-class variation and small inter-class variation. Some significant features for identification are embedded in unobvious body shape differences across pedestrians. To explore such body shape cues for cloth-changing Re-ID, we propose a Fine-grained Shape-Appearance Mutual learning framework (FSAM), a two-stream framework that learns fine-grained discriminative body shape knowledge in a shape stream and transfers it to an appearance stream to complement the cloth-unrelated knowledge in the appearance features. Specifically, in the shape stream, FSAM learns fine-grained discriminative mask with the guidance of identities and extracts fine-grained body shape features by a pose-specific multi-branch network. To complement cloth-unrelated shape knowledge in the appearance stream, dense interactive mutual learning is performed across low-level and high-level features to transfer knowledge from shape stream to appearance stream, which enables the appearance stream to be deployed independently without extra computation for mask estimation. We evaluated our method on benchmark cloth-changing Re-ID datasets and achieved the start-of-the-art performance.

Introduction: The Cloth-Changing Challenge

Person Re-Identification (Re-ID) aims to match individuals across non-overlapping camera views. While standard models have reached high accuracy, they often rely on color and clothing patterns. In long-term scenarios, individuals frequently change outfits, leading to two primary issues: high intra-class variation (the same person looks different in different clothes) and low inter-class variation (different people look similar in the same clothes).

The cloth-changing problem: high intra-class variation (red arrow) and low inter-class variation (green arrow).

Traditional body shape features, often derived from coarse human parsing masks, are typically insufficient for distinguishing between individuals with similar builds. Moreover, silhouettes are highly sensitive to pose changes, and generating masks in real-time is computationally prohibitive for large-scale deployment.

The FSAM Framework

To address these challenges, we propose the Fine-grained Shape-Appearance Mutual learning framework (FSAM). FSAM utilizes a two-stream architecture—a Shape Stream and an Appearance Stream—to extract and transfer discriminative, cloth-independent features.

1. Identity-Guided Mask Learning

Standard human parsing focuses on anatomical parts (e.g., "arm", "leg"). FSAM instead uses identity labels to refine masks, forcing the Shape Stream to capture "fine-grained" details that distinguish individuals. This is crucial because different pedestrians often share very similar overall silhouettes.

Example of similar body shapes between different pedestrians (ID 3 and ID 4), highlighting the need for fine-grained shape features.

2. Pose-Specific Multi-branch Network

Body shape appearance varies significantly with viewing angle. FSAM addresses this by clustering human keypoints into three pose categories (e.g., front, side, back) using K-means. The network then employs specific branches to extract features optimized for each pose, reducing noise from perspective shifts.

3. Dense Interactive Mutual Learning

The core of FSAM is the mutual learning process between the two streams during training:

  • Logit-level alignment: Kullback-Leibler (KL) Divergence aligns the final probability outputs.
  • Feature-level transfer: A "Dense Similarity Loss" compares feature similarity matrices across intermediate layers, effectively transferring cloth-unrelated shape knowledge into the Appearance Stream.

4. Efficient Inference

A key advantage of FSAM is that the Shape Stream is only used during training. The Appearance Stream "inherits" shape-aware knowledge, allowing it to be deployed independently. This eliminates the need for expensive mask estimation during inference.

Experimental Evaluation

FSAM was evaluated on several cloth-changing benchmarks and standard Re-ID datasets:

  • LTCC: Achieved 38.5% Rank-1 accuracy in cloth-changing settings, significantly outperforming the ResNet50 baseline (29.8%).
  • PRCC: Reached 54.5% Rank-1 accuracy, a substantial improvement over previous specialized methods.
  • Market-1501: Maintained a high 94.6% Rank-1, demonstrating that shape-awareness does not compromise performance in standard scenarios where clothes do not change.

Conclusion

FSAM effectively tackles the cloth-changing Re-ID problem by treating shape masks as a "teacher" for the appearance model. By focusing on discriminative, fine-grained shape cues and accounting for pose variations, FSAM achieves state-of-the-art results while maintaining inference efficiency. Future work may explore integrating additional cloth-independent features like facial details or hair styles to further enhance robustness.

Summary updated

Abstract

Recently, person re-identification (Re-ID) has achieved great progress. However, current methods largely depend on color appearance, which is not reliable when a person changes the clothes. Cloth-changing Re-ID is challenging since pedestrian images with clothes change exhibit large intra-class variation and small inter-class variation. Some significant features for identification are embedded in unobvious body shape differences across pedestrians. To explore such body shape cues for cloth-changing Re-ID, we propose a Fine-grained Shape-Appearance Mutual learning framework (FSAM), a two-stream framework that learns fine-grained discriminative body shape knowledge in a shape stream and transfers it to an appearance stream to complement the cloth-unrelated knowledge in the appearance features. Specifically, in the shape stream, FSAM learns fine-grained discriminative mask with the guidance of identities and extracts fine-grained body shape features by a pose-specific multi-branch network. To complement cloth-unrelated shape knowledge in the appearance stream, dense interactive mutual learning is performed across low-level and high-level features to transfer knowledge from shape stream to appearance stream, which enables the appearance stream to be deployed independently without extra computation for mask estimation. We evaluated our method on benchmark cloth-changing Re-ID datasets and achieved the start-of-the-art performance.
BibTeX
@inproceedings{hong2021fine,
  author = {Peixian Hong and Tao Wu and Ancong Wu and Xintong Han and Wei-Shi Zheng},
  title = {{Fine-Grained Shape-Appearance Mutual Learning for Cloth-Changing Person Re-Identification}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2021},
  url = {https://openreview.net/forum?id=2F8x_HaEl7w},
}
Figure