Good is Bad: Causality Inspired Cloth-debiasing for Cloth-changing Person Re-identification

Zhengwei Yang, Meng Lin, Xian Zhong, Yu Wu, Zheng WangCVPR 2023 · OpenReview

Abstract

Entangled representation of clothing and identity (ID)-intrinsic clues are potentially concomitant in conventional person Re-IDentification (ReID). Nevertheless, eliminating the negative impact of clothing on ID remains challenging due to the lack of theory and the difficulty of isolating the exact implications. In this paper, a causality-based Auto-Intervention Model, referred to as AIM, is first proposed to mitigate clothing bias for robust cloth-changing person ReID (CC-ReID). Specifically, we analyze the effect of clothing on the model inference and adopt a dual-branch model to simulate causal intervention. Progressively, clothing bias is eliminated automatically with model training. AIM is encouraged to learn more discriminative ID clues that are free from clothing bias. Extensive experiments on two standard CC-ReID datasets demonstrate the superiority of the proposed AIM over other state-of-the-art methods.

Introduction

Person Re-Identification (ReID) is the fundamental task of matching a person across different camera views. In short-term scenarios, clothing is a highly reliable signal. However, in long-term surveillance or forensic applications, people change their clothes, causing standard models to fail.

The authors argue that clothing is a "double-edged sword." Because clothing and identity are so highly correlated in training data, neural networks develop an entangled representation. They learn to treat specific clothing as a proxy for identity. This "clothing bias" prevents models from learning more robust, immutable biometric features like gait, body shape, or facial structure. This paper introduces the Auto-Intervention Model (AIM), which uses causal inference to identify and remove this bias.

Illustration of the clothing bias problem: (a) Entangled representation of clothing and ID leading to incorrect matches. (b) Clothing effects on ID classification.

Methodology

The core of the paper is the AIM, which treats the ReID task through the lens of causal inference to "subtract" clothing bias.

Dual-Branch Architecture

The model consists of two parallel branches:

  • ID Branch: A standard ReID network aimed at learning identity features.
  • Clothing Branch: Specifically designed to extract clothing features using a "pyramid matching" strategy, analyzing the image at different scales to capture shirts, pants, and shoes.

Causal Intervention

The authors utilize the concept of Total Direct Effect (TDE). In causal terms, they isolate the effect of the person's identity ($X$) on the prediction ($Y$) while removing the confounding effect of clothing ($C$). By simulating an "intervention" where the clothing-only prediction is subtracted from the entangled (ID + clothing) prediction, the ID branch is forced to focus on features that cannot be explained by clothing alone.

The overall architecture of the Auto-Intervention Model (AIM), showing the dual-branch structure and the causal intervention process.

Training and Inference

During training, Knowledge Transfer via Kullback-Leibler (KL) divergence ensures the clothing branch effectively distills the bias out of the ID branch. During inference, the clothing branch is discarded, and the debiased ID branch is used alone to extract robust features.

Experimental Results

AIM was evaluated on two primary "Cloth-Changing" (CC-ReID) benchmarks:

  • PRCC-ReID: AIM achieved a 57.9% Rank-1 accuracy, outperforming the baseline (54.4%) and previous state-of-the-art methods like IRANet (54.9%).
  • LTCC-ReID: AIM reached 40.6% Rank-1, a substantial jump over the baseline's 34.4%.

Visualization through heatmaps confirms that while baseline models focus heavily on the torso (clothing), AIM’s attention shifts toward more stable features like the head and overall body proportions. Notably, AIM achieves these results using pure RGB data, without requiring extra modalities like sketches or 3D shape modeling.

Conclusion

The paper demonstrates that "Good is Bad": highly discriminative features like a unique shirt are "good" for short-term accuracy but "bad" for long-term model robustness. By using causality as a mathematical scalpel to subtract these transient features, AIM achieves state-of-the-art performance in cloth-changing scenarios. Future work may extend this causal framework to address other confounders such as lighting and camera angle.

Summary updated

Abstract

Entangled representation of clothing and identity (ID)-intrinsic clues are potentially concomitant in conventional person Re- IDentification (ReID). Nevertheless, eliminating the negative impact of clothing on ID remains challenging due to the lack of theory and the difficulty of isolating the exact implications. In this paper, a causality-based Auto-Intervention Model, referred to as AIM 1 1 Codes will publicly available at https://github.com/BoomShakaY/AIM-CCReID., is first proposed to mitigate clothing bias for robust cloth-changing person ReID (CC-ReID). Specifically, we analyze the effect of clothing on the model inference and adopt a dual-branch model to simulate causal intervention. Progressively, clothing bias is eliminated automatically with model training. AIM is encouraged to learn more discriminative ID clues that are free from clothing bias. Extensive experiments on two standard CC-ReID datasets demonstrate the superiority of the proposed AIM over other state-of-the-art methods.
BibTeX
@inproceedings{yang2023good,
  author = {Zhengwei Yang and Meng Lin and Xian Zhong and Yu Wu and Zheng Wang},
  title = {{Good is Bad: Causality Inspired Cloth-debiasing for Cloth-changing Person Re-identification}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
  url = {https://openreview.net/forum?id=zpfJPeQxKl},
}
Figure