Modality-Adaptive Mixup and Invariant Decomposition for RGB-Infrared Person Re-Identification

Zhipeng Huang, Jiawei Liu, Liang Li, Kecheng Zheng, Zheng-Jun Zha

Introduction

Person Re-Identification (Re-ID) is a critical component of modern surveillance systems, enabling the tracking of individuals across non-overlapping camera views. While significant progress has been made in visible-light (RGB) Re-ID, real-world applications often require nighttime monitoring where cameras switch to Infrared (IR) mode. This transition introduces a significant "modality gap" due to the distinct physical properties of RGB (color and texture) and IR (thermal radiation and shape) data. The Modality-Adaptive Mixup and Invariant Decomposition (MID) framework addresses this challenge by bridging the gap at both the data and architectural levels.

The Modality Gap Challenge

The discrepancy between RGB and IR images manifests at two primary levels:

  1. Pixel-level Discrepancy: The raw data distributions are fundamentally different; RGB captures visible light while IR captures thermal signatures, resulting in grayscale-like images with different noise profiles.
  2. Feature-level Discrepancy: Standard neural networks struggle to extract a unified representation that remains invariant across spectrums, as the features learned for color textures do not translate to thermal shapes.

Traditional approaches, such as Generative Adversarial Networks (GANs), attempt to translate IR to RGB. However, these models are often difficult to stabilize and can introduce artifacts or "noise" that degrade identification accuracy. MID proposes a more stable alternative through intelligent augmentation and layer decomposition.

Methodology

1. Modality-Adaptive Mixup (MAM)

To bridge the gap without the instability of GANs, the authors introduce the Modality-Adaptive Mixup (MAM) scheme. Standard "Mixup" techniques apply a uniform linear blend across an entire image, which often leads to "manifold intrusion"—a phenomenon where the resulting image becomes blurry and loses identity-defining features.

MAM utilizes Deep Reinforcement Learning to solve this. An Actor-Critic agent analyzes pairs of RGB and IR images to determine optimal interpolation ratios for specific local regions (horizontal partitions). This localized, dynamic approach ensures that the generated "mixed" images preserve structural integrity and identity information, serving as a crucial intermediate "stepping stone" in the latent space between the two modalities.

2. Modality-Adaptive Convolution Decomposition (MACD)

Complementing the data augmentation is a structural innovation: Modality-Adaptive Convolution Decomposition (MACD). Recognizing that a single set of convolutional filters is often insufficient for cross-modality tasks, the authors decompose standard layers into two components:

  • Modality-specific basis layers: These act as specialized "dictionaries" that learn the unique characteristics of RGB, IR, and the newly created Mixed modalities separately.
  • Modality-shared coefficient layer: This layer extracts common semantic features, such as body shape and proportions, which are consistent across all modalities.

This architecture allows the model to specialize in individual spectrums while maintaining a shared understanding of human identity, avoiding the parameter bloat associated with traditional two-stream networks.

Experimental Evaluation

The MID framework was evaluated on the two primary benchmarks in the field:

  • RegDB: Consisting of 412 pedestrians with 10 RGB and 10 thermal images each.
  • SYSU-MM01: The largest benchmark, featuring 491 identities captured by 4 RGB and 2 IR cameras.

Key Results

MID outperformed all previous state-of-the-art (SOTA) methods on both datasets:

  • RegDB: Achieved 87.45% Rank-1 accuracy, a significant ~7% improvement over the previous best method (NFS).
  • SYSU-MM01: Achieved 60.27% Rank-1 accuracy in "All-search" mode, outperforming the previous SOTA by over 3%.

Qualitative analysis confirmed that the RL-driven mixup produced clearer, more recognizable samples compared to the "ghosting" effects seen in vanilla global mixup.

Discussion and Future Work

MID effectively addresses the modality gap through a combination of intelligent data synthesis and adaptive architectural decomposition. By partitioning images and using an RL agent, the model handles occlusions and lighting variations more robustly than global methods.

However, some limitations remain. The inclusion of an Actor-Critic agent adds training complexity (approximately 6 hours on SYSU-MM01), and the current formulation uses a "one-step" Markov Decision Process. Future research may explore multi-step reinforcement learning for iterative image refinement and the application of MACD to other cross-modality domains, such as Sketch-to-Photo or Depth-to-RGB Re-ID.

Summary updated

BibTeX
@inproceedings{huang2022modality,
  author = {Zhipeng Huang and Jiawei Liu and Liang Li and Kecheng Zheng and Zheng-Jun Zha},
  title = {{Modality-Adaptive Mixup and Invariant Decomposition for RGB-Infrared Person Re-Identification}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2022},
  doi = {10.1609/aaai.v36i1.19987},
}
Figure