Abstract
Visible-Infrared Person Re-Identification (VI-ReID) is a critical task for 24/7 intelligent surveillance, yet it is hindered by the significant "modality gap" between visible (VIS) and near-infrared (NIR) spectra. Standard training losses, such as Softmax, often fail to explicitly account for these discrepancies, leading to suboptimal feature alignment. This paper introduces Spectral-Aware Softmax (SA-Softmax), a novel loss function designed to force cross-spectral alignment. By optimizing samples against prototypes from the opposite spectrum and employing a feature masking strategy, SA-Softmax significantly boosts retrieval accuracy in low-light scenarios, achieving state-of-the-art results on major benchmarks.
Addressing the Modality Gap
VI-ReID systems must match pedestrians across different camera types: visible cameras during the day and NIR cameras at night. The fundamental challenge is that the same person appears differently across these spectra due to varying wavelengths and lighting conditions.
Existing approaches often rely on standard Softmax loss, which treats VIS and NIR images of the same identity as a single class. However, Softmax is inherently modality-blind; it does not provide an explicit penalty for the distance between VIS and NIR features of the same person. As a result, a query image may be closer to a different person in the same spectrum (intra-modality inter-class) than to the correct person in a different spectrum (cross-modality intra-class).
The SA-Softmax Framework
The core of the proposed method is the Spectral-Aware Softmax (SA-Softmax), which rethinks the classification process to bridge the spectral divide. The framework consists of three primary innovations:
- Asynchronous Modality Prototypes: Unlike standard methods that use a single prototype per identity, SA-Softmax maintains two distinct prototypes: $W^v$ for visible and $W^n$ for infrared. During training, a VIS sample is optimized to align with its corresponding NIR prototype, and vice versa. This "cross-spectral pulling" forces the two distributions into a shared latent space.
- Feature Mask (FM): To prevent the model from simply clustering same-modality features, the Feature Mask excludes the same-modality prototype from the loss calculation. This ensures the optimization gradient is entirely focused on bridging the spectral gap.
- Absolute-Similarity Term (AST): While Softmax is a relative loss, AST adds an explicit penalty to maximize the absolute cosine similarity between a sample and its target cross-modal prototype, leading to tighter, more compact identity clusters.
Experimental Results
The authors evaluated SA-Softmax on two primary VI-ReID benchmarks, demonstrating its effectiveness as a "plug-and-play" enhancement for standard backbones like ResNet-50.
- RegDB: The method achieved a 95.2% Rank-1 accuracy and 88.7% mAP in Visible-to-Thermal search, significantly outperforming previous state-of-the-art (SOTA) models like CAJ and MCLNet.
- SYSU-MM01: In the challenging "All-Search" mode, SA-Softmax reached 71.0% Rank-1 accuracy. While the improvement here was slightly less pronounced than on RegDB due to the dataset's complex pose and background variations, it still established a new performance baseline.
Visualization via t-SNE plots confirms that while standard Softmax leaves VIS and NIR features in separate clusters, SA-Softmax successfully merges them into unified, compact identity representations.
Conclusion
SA-Softmax addresses the modality bias inherent in standard classification losses for VI-ReID. By utilizing asynchronous prototypes and cross-modality optimization, the framework effectively bridges the spectral gap. Future work may involve integrating SA-Softmax with advanced data augmentation or adversarial training to further mitigate the impact of extreme pose and viewpoint variations in complex environments.
Summary updated