Abstract
Recently, person re-identification (Re-ID) has achieved great progress. However, current methods largely depend on color appearance, which is not reliable when a person changes the clothes. Cloth-changing Re-ID is challenging since pedestrian images with clothes change exhibit large intra-class variation and small inter-class variation. Some significant features for identification are embedded in unobvious body shape differences across pedestrians. To explore such body shape cues for cloth-changing Re-ID, we propose a Fine-grained Shape-Appearance Mutual learning framework (FSAM), a two-stream framework that learns fine-grained discriminative body shape knowledge in a shape stream and transfers it to an appearance stream to complement the cloth-unrelated knowledge in the appearance features. Specifically, in the shape stream, FSAM learns fine-grained discriminative mask with the guidance of identities and extracts fine-grained body shape features by a pose-specific multi-branch network. To complement cloth-unrelated shape knowledge in the appearance stream, dense interactive mutual learning is performed across low-level and high-level features to transfer knowledge from shape stream to appearance stream, which enables the appearance stream to be deployed independently without extra computation for mask estimation. We evaluated our method on benchmark cloth-changing Re-ID datasets and achieved the start-of-the-art performance.
Introduction: The Cloth-Changing Challenge
Person Re-Identification (Re-ID) aims to match individuals across non-overlapping camera views. While standard models have reached high accuracy, they often rely on color and clothing patterns. In long-term scenarios, individuals frequently change outfits, leading to two primary issues: high intra-class variation (the same person looks different in different clothes) and low inter-class variation (different people look similar in the same clothes).
Traditional body shape features, often derived from coarse human parsing masks, are typically insufficient for distinguishing between individuals with similar builds. Moreover, silhouettes are highly sensitive to pose changes, and generating masks in real-time is computationally prohibitive for large-scale deployment.
The FSAM Framework
To address these challenges, we propose the Fine-grained Shape-Appearance Mutual learning framework (FSAM). FSAM utilizes a two-stream architecture—a Shape Stream and an Appearance Stream—to extract and transfer discriminative, cloth-independent features.
1. Identity-Guided Mask Learning
Standard human parsing focuses on anatomical parts (e.g., "arm", "leg"). FSAM instead uses identity labels to refine masks, forcing the Shape Stream to capture "fine-grained" details that distinguish individuals. This is crucial because different pedestrians often share very similar overall silhouettes.
2. Pose-Specific Multi-branch Network
Body shape appearance varies significantly with viewing angle. FSAM addresses this by clustering human keypoints into three pose categories (e.g., front, side, back) using K-means. The network then employs specific branches to extract features optimized for each pose, reducing noise from perspective shifts.
3. Dense Interactive Mutual Learning
The core of FSAM is the mutual learning process between the two streams during training:
- Logit-level alignment: Kullback-Leibler (KL) Divergence aligns the final probability outputs.
- Feature-level transfer: A "Dense Similarity Loss" compares feature similarity matrices across intermediate layers, effectively transferring cloth-unrelated shape knowledge into the Appearance Stream.
4. Efficient Inference
A key advantage of FSAM is that the Shape Stream is only used during training. The Appearance Stream "inherits" shape-aware knowledge, allowing it to be deployed independently. This eliminates the need for expensive mask estimation during inference.
Experimental Evaluation
FSAM was evaluated on several cloth-changing benchmarks and standard Re-ID datasets:
- LTCC: Achieved 38.5% Rank-1 accuracy in cloth-changing settings, significantly outperforming the ResNet50 baseline (29.8%).
- PRCC: Reached 54.5% Rank-1 accuracy, a substantial improvement over previous specialized methods.
- Market-1501: Maintained a high 94.6% Rank-1, demonstrating that shape-awareness does not compromise performance in standard scenarios where clothes do not change.
Conclusion
FSAM effectively tackles the cloth-changing Re-ID problem by treating shape masks as a "teacher" for the appearance model. By focusing on discriminative, fine-grained shape cues and accounting for pose variations, FSAM achieves state-of-the-art results while maintaining inference efficiency. Future work may explore integrating additional cloth-independent features like facial details or hair styles to further enhance robustness.
Summary updated