RLE: A Unified Perspective of Data Augmentation for Cross-Spectral Re-identification

Lei Tan, Yukang Zhang, Keke Han, Pingyang Dai, Yan Zhang, Yongjian Wu, Rongrong Ji

Introduction

In modern intelligent surveillance systems, cameras frequently transition between visible (VIS) light during daylight hours and near-infrared (NIR) or thermal infrared (IR) at night. This transition introduces a significant challenge for person re-identification (Re-ID): matching the identity of an individual across different spectral modalities. This task, known as Cross-Spectral Re-ID, is complicated by the "modality discrepancy," where the visual appearance of clothing and skin changes drastically depending on the light spectrum used for capture.

To address this, researchers have introduced Random Linear Enhancement (RLE), a unified data augmentation framework that bridges the gap between visible and infrared images. By modeling the physical relationship between spectra as a set of local linear transformations, RLE achieves state-of-the-art performance without the computational burden or instability associated with generative models.

The Modality Discrepancy Problem

The core difficulty in Cross-Spectral Re-ID lies in the fact that pixel values in VIS and IR images do not share a simple one-to-one mapping. A red garment in a color photo might appear as bright white or dark gray in an infrared photo, depending on the material's reflective properties. Historically, researchers have approached this problem using two primary strategies:

  1. Generative Adversarial Networks (GANs): These models attempt to translate IR images into the VIS domain (or vice versa). While conceptually appealing, GANs are computationally expensive, difficult to train, and often introduce artifacts that can degrade Re-ID accuracy.
  2. Heuristic Augmentations: Simpler methods, such as converting images to grayscale or randomly swapping color channels, provide some robustness. However, these techniques are often ad-hoc and lack a grounding in the actual physics of light-surface interaction.

RLE seeks a middle ground by providing a mathematically grounded yet computationally efficient way to simulate spectral variations.

Methodology: Random Linear Enhancement

The RLE framework is built upon the Lambertian reflection model, which describes how surfaces reflect light. The authors observe that, at a pixel level, the transformation from VIS to IR can be approximated as a linear transformation. However, because a person is composed of various materials (e.g., skin, cotton, denim, polyester), the linear coefficients vary across different parts of the image. This results in a complex, non-linear discrepancy at the global image level.

To simulate this phenomenon, RLE employs two complementary components:

Moderate Random Linear Enhancement (MRLE)

MRLE targets the global style of the image. It generates a new version of the image by mixing the Red, Green, and Blue channels using random weights ($\lambda$). A key innovation here is the use of a U-shaped Beta distribution for sampling these weights. Unlike uniform distributions, the U-shaped distribution prioritizes "extreme" variations (e.g., nearly pure color channels). This forces the neural network to become invariant to significant color shifts rather than just learning to handle "average" brownish-gray mixtures.

Radical Random Linear Enhancement (RRLE)

RRLE addresses the local nature of material reflections. It selects random rectangular patches within the image and applies a random linear scaling factor ($\alpha$) to them. This mimics the way different materials on a person's body react differently to infrared light, effectively teaching the model to focus on structural identity features rather than local brightness or contrast.

Experimental Evaluation

The authors evaluated RLE on two primary benchmarks for cross-spectral Re-ID:

  • SYSU-MM01: A large-scale dataset featuring over 22,000 visible and 11,000 infrared images across 491 identities.
  • RegDB: A dataset of 412 identities captured by dual-camera systems, providing perfectly aligned visible and thermal image pairs.

Key Results

RLE demonstrated significant performance gains over existing state-of-the-art (SOTA) methods:

  • SYSU-MM01: Achieved 75.4% Rank-1 accuracy, outperforming the previous leading method (DEEN) by 0.7%.
  • RegDB: Reached 92.8% Rank-1 accuracy in Visible-to-Infrared search mode.
  • Architecture Agnostic: RLE is not limited to standard Convolutional Neural Networks (CNNs). When applied to Vision Transformers (ViT), it boosted Rank-1 accuracy by 4.2%.
  • Efficiency: Unlike GAN-based approaches, RLE adds virtually zero computational overhead during training and requires no additional parameters during inference.

Discussion and Insights

The success of RLE stems from its "Local Linear" perspective. By treating the complex modality gap as a collection of simple, local linear shifts, the framework provides a theoretical "umbrella" that unifies existing heuristic tricks like grayscale conversion and channel swapping. Furthermore, the use of U-shaped sampling ensures that the model is exposed to challenging cases during training, pushing its decision boundaries further than standard augmentation techniques.

Conclusion and Future Work

RLE provides a simple, effective, and physically grounded solution to the cross-spectral Re-ID problem. While highly effective on standard benchmarks, the authors note that the Lambertian model assumes relatively clean reflections. Future research may explore how these linear relationships hold up in extreme weather conditions, such as heavy rain or fog, where light scattering becomes more complex. Additionally, the "local linear" perspective holds promise for other computer vision tasks involving domain adaptation and image-to-image translation.

Summary updated

BibTeX
@inproceedings{tan2024rle,
  author = {Lei Tan and Yukang Zhang and Keke Han and Pingyang Dai and Yan Zhang and Yongjian Wu and Rongrong Ji},
  title = {{RLE: A Unified Perspective of Data Augmentation for Cross-Spectral Re-identification}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
  url = {https://openreview.net/forum?id=Ok6jSSxzfj},
}
Figure