Introduction
Cross-modality person re-identification (re-ID) is a critical task for 24/7 surveillance, requiring the matching of standard color (RGB) daytime images with infrared (IR) nighttime images. The fundamental challenge lies in the modality discrepancy: RGB and IR features of the same individual often appear more different to a model than features of different individuals within the same modality.
Standard approaches typically employ a "modality-agnostic proxy"—a single weight vector representing an identity across all modalities. However, this creates a "relay effect." As shown in Figure 1, while both RGB and IR features are pulled toward a central shared proxy, they are not explicitly forced toward each other. This allows a significant feature gap to persist, leading to poor cross-modal retrieval performance.
Proposed Method: MAUM
To address these limitations, the authors introduce Memory-Augmented Unidirectional Metrics (MAUM). The framework consists of two core innovations designed to bridge the modality gap directly.
Unidirectional Metrics
Instead of a shared proxy, MAUM learns two Modality-Specific Proxies (MS-Proxies) for every identity—one for RGB and one for IR. During training, the model employs a "unidirectional" metric: IR features are pulled toward the corresponding RGB proxy, and RGB features are pulled toward the IR proxy. By removing the "middle-man" shared proxy, the model is forced to align features across the modality boundary.
Memory-Based Augmentation
MAUM leverages "model drift"—the slight changes in network weights across training iterations—as a form of data augmentation. Historical versions of the MS-Proxies are stored in a memory bank. These historical proxies serve as diverse "hard positive" references. Proxies from earlier iterations, which are typically further from the current modality boundary, exert a stronger pulling force, enhancing the robustness of the learned features.
Experimental Evaluation
The researchers evaluated MAUM on two primary benchmarks:
- SYSU-MM01: A large-scale dataset with 491 identities across 4 RGB and 2 IR cameras.
- RegDB: A dual-camera dataset containing 412 identities with balanced RGB and IR samples.
The framework was also tested under "modality-imbalanced" conditions to simulate real-world scenarios where IR data might be extremely scarce (e.g., only one IR image per person).
Key Results and Insights
MAUM achieved state-of-the-art performance on both benchmarks:
- SYSU-MM01: Reached 71.68% Rank-1 accuracy and 68.79% mAP.
- RegDB: Achieved 87.87% Rank-1 accuracy for visible-to-IR search.
A significant finding is that the standard practice of using a single identity proxy actually hinders cross-modal alignment. Furthermore, while most memory-bank methods (like MoCo) use momentum to smooth out drift, MAUM demonstrates that embracing this drift can provide valuable hard-positive targets. In extreme imbalance tests, MAUM maintained 25.4% Rank-1 accuracy where the baseline dropped to 13.2%, nearly doubling performance in data-poor conditions.
Conclusion
MAUM effectively mitigates modality discrepancy by replacing shared proxies with memory-augmented unidirectional metrics. While the approach introduces some computational overhead due to memory bank management, it significantly improves retrieval accuracy and robustness. Future work may focus on optimizing memory efficiency and exploring the integration of part-based features into simpler global-feature architectures.
Summary updated