Single-modality self-supervised information mining for cross-modality person re-identification

Ancong Wu, Chengzhi Lin, Wei‐Shi ZhengJournal of Image and Graphics 2022 · DOI

Journal Cover of the publication source.

Abstract

In the field of intelligent surveillance, person re-identification (Re-ID) is a fundamental task for matching individuals across non-overlapping camera views. While visible-light matching has achieved high performance, cross-modality matching between visible and infrared (IR) images remains a significant challenge. This paper presents a novel approach using single-modality self-supervised information mining to bridge the "modality gap" and overcome the scarcity of labeled cross-modal data. By leveraging abundant labeled visible-light data, the proposed method extracts spectrum-invariant features to guide the learning of cross-modal matching models.

The Challenge: Bridging the Modality Gap

Matching a person's identity between standard color images and low-light infrared images is difficult due to two primary hurdles:

  1. Visual Discrepancy: Significant differences in spectral imaging between visible and infrared light create a wide modality gap. For instance, a red shirt in daylight looks completely different in a thermal scan.
  2. Data Scarcity: Manually labeling cross-modality identities is labor-intensive and often difficult for human annotators, leading to a lack of high-quality training data.
Conceptual illustration of the data bottleneck and limitations in image processing.

Methodology: Mining Spectrum-Invariant Features

To address these issues, the researchers introduced a two-pronged strategy focused on extracting features that remain constant regardless of the camera type.

Random Single-Channel Mask (RSCM)

The authors developed RSCM, a data augmentation technique that randomly masks two of the three RGB channels in visible-light images. By forcing the model to recognize individuals using only a single channel (mimicking the single-channel nature of IR), the network learns to prioritize structural and textural cues—such as clothing patterns and body shape—over specific color values.

Dual-Model Mutual Learning

The framework employs two models trained simultaneously:

  • Three-channel model: Processes full RGB data.
  • Single-channel model: Processes masked data. Through a mutual learning loss (KL divergence), these models exchange knowledge. The single-channel model learns identity-rich features from the RGB model, while the RGB model becomes more robust by learning to ignore color-specific noise that does not translate across modalities.

Two-Stage Training Strategy

The process begins with pre-training on large-scale labeled RGB data (including synthetic datasets like UnrealPerson) to mine robust cross-spectral self-supervised information. This is followed by supervised fine-tuning on specific cross-modality datasets to bridge the final domain gap between RGB and IR.

Experimental Results

The proposed method was evaluated on three major benchmarks: SYSU-MM01, RGBNT201, and RegDB.

  • RGBNT201: The approach achieved a significant performance boost, with the mean Average Precision (mAP) increasing by nearly 5% over previous state-of-the-art methods.
  • SYSU-MM01: In the "All Search" mode, the model reached a Rank-1 accuracy of 73.0% and an mAP of 70.3%, outperforming more complex specialized architectures.
  • RegDB: The system achieved 89.9% Rank-1 accuracy for visible-to-thermal matching tasks.

Conclusion

By shifting the focus from color-dependent features to spectrum-invariant structural information, this research provides a scalable solution to the cross-modality Re-ID problem. The use of synthetic data and self-supervised mining effectively reduces the reliance on expensive manual cross-modal labeling, paving the way for more robust surveillance systems capable of operating across diverse lighting conditions.

Summary updated

BibTeX
@article{wu2022single,
  author = {Ancong Wu and Chengzhi Lin and Wei‐Shi Zheng},
  title = {{Single-modality self-supervised information mining for cross-modality person re-identification}},
  journal = {Journal of Image and Graphics},
  year = {2022},
  doi = {10.11834/jig.211050},
}
Figure