Towards Grand Unified Representation Learning for Unsupervised Visible-Infrared Person Re-Identification

Bin Yang, Jun Chen, Mang Ye

Introduction

Person Re-Identification (ReID) is a critical computer vision task focused on identifying the same individual across non-overlapping camera views. While standard ReID typically utilizes visible-light (RGB) imagery, practical 24-hour surveillance requires Visible-Infrared ReID (VI-ReID). This modality allows systems to switch to infrared (IR) sensors in low-light conditions.

The most challenging frontier in this field is Unsupervised VI-ReID (USL-VI-ReID), which aims to learn robust features without the need for expensive manual labeling. This paper introduces the Grand Unified Representation (GUR) framework, a novel approach designed to overcome the inherent complexities of matching identities across disparate imaging modalities without supervision.

The Challenge of Hierarchical Discrepancy

Unsupervised VI-ReID faces a unique "double-whammy" of challenges known as hierarchical discrepancy. As illustrated in the figure below, the model must simultaneously contend with variations within a single modality and the massive gap between modalities.

Hierarchical discrepancies in VI-ReID leading to clustering inconsistencies and ambiguous label association.
  1. Intra-modality variation: Differences in viewpoint, illumination, and background between cameras of the same type (e.g., two different RGB cameras).
  2. Inter-modality discrepancy: The significant visual gap between color RGB images and monochromatic IR images caused by different imaging wavelengths.

Existing unsupervised methods often fail to distinguish between these levels, leading to "clustering inconsistencies" (where a single person is split into multiple IDs) and "ambiguous label association" (where the model fails to link an RGB cluster to its corresponding IR cluster).

The Grand Unified Representation (GUR) Framework

To address these discrepancies, the authors propose the GUR framework. GUR adopts a bottom-up strategy to progressively bridge the gaps between cameras and modalities.

Overview of the Grand Unified Representation (GUR) framework, including Bottom-up Domain Learning, Cross-memory Association Embedding, and Cross-modality Label Unification.

Key Methodological Components

  • Bottom-up Domain Learning (BD): Instead of attempting to match RGB and IR images immediately, the model follows a three-stage training hierarchy: intra-camera (learning features within a single camera), inter-camera (matching across cameras of the same modality), and finally inter-modality (matching RGB to IR).
  • Cross-memory Association Embedding (CAE): This module calculates the "association probability" between an image and all existing clusters in a specific domain. By concatenating these probabilities across all domains, the model creates a unified representation. This approach relies on the insight that while raw pixels change between RGB and IR, a person's relative relationship to other clusters (their association distribution) remains stable.
  • Cross-modality Label Unification (CLU): This module ensures that identity labels are consistent across modalities. It uses a heterogeneous affinity matrix to propagate labels between RGB and IR, while a homogeneous structure matrix "smooths" these labels to maintain the integrity of clusters already established within a single modality.

Experimental Results

The GUR framework was evaluated on two standard benchmarks, SYSU-MM01 and RegDB, setting a new state-of-the-art for unsupervised VI-ReID:

  • SYSU-MM01: GUR achieved 63.51% Rank-1 accuracy, outperforming the previous best unsupervised method (ADCA) by approximately 18%.
  • RegDB: The framework reached 73.91% Rank-1 accuracy, a 30% improvement over previous unsupervised state-of-the-art methods.
  • Comparison to Supervised Learning: Remarkably, GUR's unsupervised performance surpassed several established supervised methods (such as Zero-Padding and AGW), demonstrating that a structured hierarchical approach can be more effective than basic supervised learning.

Conclusion and Future Work

The Grand Unified Representation framework demonstrates that hierarchical discrepancies in VI-ReID are best solved through a progressive, bottom-up approach. By treating distribution as identity through the CAE module and smoothing cross-modality labels with CLU, GUR achieves massive performance leaps.

Future work will focus on closing the remaining gap between GUR and the most advanced supervised models. The authors suggest that exploring transformer-based architectures for global and part-based feature association could further refine label accuracy and cross-modality matching efficiency.

Summary updated

Abstract

Unsupervised learning visible-infrared person re-identification (USL-VI-ReID) is an extremely important and challenging task, which can alleviate the issue of expensive cross-modality annotations. Existing works focus on handling the cross-modality discrepancy under unsupervised conditions. However, they ignore the fact that USL-VI-ReID is a cross-modality retrieval task with the hierarchical discrepancy, i.e., camera variation and modality discrepancy, resulting in clustering inconsistencies and ambiguous cross-modality label association. To address these issues, we propose a hierarchical framework to learn grand unified representation (GUR) for USL-VI-ReID. The grand unified representation lies in two aspects: 1) GUR adopts a bottom-up domain learning strategy with a cross-memory association embedding module to explore the information of hierarchical domains, i.e., intra-camera, inter-camera, and inter-modality domains, learning a unified and robust representation against hierarchical discrepancy. 2) To unify the identities of the two modalities, we develop a cross-modality label unification module that constructs a cross-modality affinity matrix as a bridge for propagating labels between two modalities. Then, we utilize the homogeneous structure matrix to smooth the propagated labels, ensuring that the label structure within one modality remains unchanged. Extensive experiments demonstrate that our GUR framework significantly outperforms existing USL-VI-ReID methods, and even surpasses some supervised counterparts.
BibTeX
@inproceedings{yang2023towards,
  author = {Bin Yang and Jun Chen and Mang Ye},
  title = {{Towards Grand Unified Representation Learning for Unsupervised Visible-Infrared Person Re-Identification}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
  doi = {10.1109/iccv51070.2023.01016},
}
Figure