Learning Memory-Augmented Unidirectional Metrics for Cross-modality Person Re-identification

Jialun Liu, Yifan Sun, Feng Zhu, Hongbin Pei, Yi Yang, Wenhui Li

Introduction

Cross-modality person re-identification (re-ID) is a critical task for 24/7 surveillance, requiring the matching of standard color (RGB) daytime images with infrared (IR) nighttime images. The fundamental challenge lies in the modality discrepancy: RGB and IR features of the same individual often appear more different to a model than features of different individuals within the same modality.

Standard approaches typically employ a "modality-agnostic proxy"—a single weight vector representing an identity across all modalities. However, this creates a "relay effect." As shown in Figure 1, while both RGB and IR features are pulled toward a central shared proxy, they are not explicitly forced toward each other. This allows a significant feature gap to persist, leading to poor cross-modal retrieval performance.

Comparison between a modality-agnostic proxy (baseline) and the proposed multiple MS-specific proxies (MAUM).

Proposed Method: MAUM

To address these limitations, the authors introduce Memory-Augmented Unidirectional Metrics (MAUM). The framework consists of two core innovations designed to bridge the modality gap directly.

Unidirectional Metrics

Instead of a shared proxy, MAUM learns two Modality-Specific Proxies (MS-Proxies) for every identity—one for RGB and one for IR. During training, the model employs a "unidirectional" metric: IR features are pulled toward the corresponding RGB proxy, and RGB features are pulled toward the IR proxy. By removing the "middle-man" shared proxy, the model is forced to align features across the modality boundary.

Memory-Based Augmentation

MAUM leverages "model drift"—the slight changes in network weights across training iterations—as a form of data augmentation. Historical versions of the MS-Proxies are stored in a memory bank. These historical proxies serve as diverse "hard positive" references. Proxies from earlier iterations, which are typically further from the current modality boundary, exert a stronger pulling force, enhancing the robustness of the learned features.

The overall architecture of the Memory-Augmented Unidirectional Metrics (MAUM) framework.

Experimental Evaluation

The researchers evaluated MAUM on two primary benchmarks:

  • SYSU-MM01: A large-scale dataset with 491 identities across 4 RGB and 2 IR cameras.
  • RegDB: A dual-camera dataset containing 412 identities with balanced RGB and IR samples.

The framework was also tested under "modality-imbalanced" conditions to simulate real-world scenarios where IR data might be extremely scarce (e.g., only one IR image per person).

Key Results and Insights

MAUM achieved state-of-the-art performance on both benchmarks:

  • SYSU-MM01: Reached 71.68% Rank-1 accuracy and 68.79% mAP.
  • RegDB: Achieved 87.87% Rank-1 accuracy for visible-to-IR search.

A significant finding is that the standard practice of using a single identity proxy actually hinders cross-modal alignment. Furthermore, while most memory-bank methods (like MoCo) use momentum to smooth out drift, MAUM demonstrates that embracing this drift can provide valuable hard-positive targets. In extreme imbalance tests, MAUM maintained 25.4% Rank-1 accuracy where the baseline dropped to 13.2%, nearly doubling performance in data-poor conditions.

Conclusion

MAUM effectively mitigates modality discrepancy by replacing shared proxies with memory-augmented unidirectional metrics. While the approach introduces some computational overhead due to memory bank management, it significantly improves retrieval accuracy and robustness. Future work may focus on optimizing memory efficiency and exploring the integration of part-based features into simpler global-feature architectures.

Summary updated

Abstract

This paper tackles the cross-modality person reidentification (re-ID) problem by suppressing the modality discrepancy. In cross-modality re-ID, the query and gallery images are in different modalities. Given a training identity, the popular deep classification baseline shares the same proxy (i.e., a weight vector in the last classification layer) for two modalities. We find that it has considerable tolerance for the modality gap, because the shared proxy acts as an intermediate relay between two modalities. In response, we propose a Memory-Augmented Unidirectional Metric (MAUM) learning method consisting of two novel designs, i.e., unidirectional metrics, and memory-based augmentation. Specifically, MAUM first learns modality-specific proxies (MS-Proxies) independently under each modality. Afterward, MAUM uses the already-learned MS-Proxies as the static references for pulling close the features in the counterpart modality. These two unidirectional metrics (IR image to RGB proxy and RGB image to IR proxy) jointly alleviate the relay effect and benefit cross-modality association. The cross-modality association is further enhanced by storing the MS-Proxies into memory banks to increase the reference diversity. Importantly, we show that MAUM improves cross-modality re-ID under the modality-balanced setting and gains extra robustness against the modality imbalance problem. Extensive experiments on SYSU-MM01 and RegDB datasets demonstrate the superiority of MAUM over the state-of-the-art. The code will be available.
BibTeX
@inproceedings{liu2022learning,
  author = {Jialun Liu and Yifan Sun and Feng Zhu and Hongbin Pei and Yi Yang and Wenhui Li},
  title = {{Learning Memory-Augmented Unidirectional Metrics for Cross-modality Person Re-identification}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2022},
  doi = {10.1109/cvpr52688.2022.01876},
}
Figure