RGB-Infrared Cross-Modality Person Re-identification

Ancong Wu, Wei-Shi Zheng, Hong-Xing Yu, Shaogang Gong, Jianhuang LaiICCV 2017 · DOI

Introduction

Person re-identification (Re-ID) is the task of identifying the same individual across different non-overlapping camera views. While standard Re-ID has seen significant progress, most existing models rely on RGB-to-RGB matching. This approach fails in 24-hour surveillance scenarios: at night, RGB cameras produce uninformative, noisy images, prompting systems to switch to Infrared (IR) mode.

Comparison of RGB and IR camera captures across day and night.

The primary challenge in this context is the modality gap. RGB and IR images are heterogeneous; RGB images contain three channels of color information, whereas IR images are single-channel and capture different light wavelengths. Consequently, traditional appearance cues, such as clothing color, are virtually useless for cross-modality matching. This research aims to bridge this gap by identifying shared features that persist across both spectrums.

Methodology

The authors investigate how different neural network architectures handle the domain shift between RGB and IR, comparing three standard structures: one-stream (shared parameters), two-stream (domain-specific early layers), and asymmetric fully connected (shared features with separate final layers).

The core contribution is a technique called Deep Zero-Padding. Rather than manually designing complex multi-stream architectures, the authors utilize a one-stream network with modified inputs:

  • For RGB: The image is converted to grayscale and placed in the first channel of a two-channel input, with the second channel padded with zeros.
  • For IR: The IR image is placed in the second channel, with the first channel padded with zeros.

This simple "trick" provides a clear signal to the network regarding the modality it is processing. Mathematically, this allows a standard one-stream network to evolve "domain-specific nodes" automatically during training, effectively learning to apply different filters to RGB versus IR images in the early layers.

The SYSU-MM01 Dataset

To facilitate research in this area, the authors introduced SYSU-MM01, the first large-scale benchmark for RGB-IR cross-modality Re-ID. Unlike existing single-modality datasets like Market-1501, SYSU-MM01 provides the necessary data to train and test cross-spectrum matching.

Sample images from the SYSU-MM01 dataset across six different camera views.

The dataset includes 491 identities captured by 6 cameras (4 RGB, 2 IR), totaling 287,628 RGB images and 15,792 IR images. It encompasses both indoor and outdoor scenes with varying poses and viewpoints.

Key Results and Insights

Experiments on SYSU-MM01 demonstrate that the RGB-IR task is significantly more challenging than standard Re-ID. However, the proposed Deep Zero-Padding method showed clear gains:

  • Performance: On the "all-search" task, deep zero-padding achieved 14.80% Rank-1 accuracy, outperforming standard one-stream (12.04%), two-stream (11.65%), and asymmetric FC (9.30%) structures.
  • Domain Indicator Effect: Zero-padding acts as a persistent domain indicator that propagates through the network, helping the model alleviate differences in gradient orientations and exposure between modalities.
  • Failure of Handcrafted Features: Traditional features like LOMO or HOG, which perform well in RGB Re-ID, often stay below 10% accuracy in the RGB-IR task, highlighting the need for deep learning to find abstract, structural similarities.

Conclusion

While the proposed method shows improvement, a 14.8% Rank-1 accuracy indicates that RGB-IR matching remains an open and challenging problem. Future work should focus on more sophisticated domain adaptation techniques for the IR spectrum and improving robustness to extreme viewpoint changes where color information is absent.

Summary updated

BibTeX
@inproceedings{wu2017rgb,
  author = {Ancong Wu and Wei-Shi Zheng and Hong-Xing Yu and Shaogang Gong and Jianhuang Lai},
  title = {{RGB-Infrared Cross-Modality Person Re-identification}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2017},
  doi = {10.1109/iccv.2017.575},
}
Figure