Spectral Aware Softmax for Visible-Infrared Person Re-Identification

Lei Tan, Pingyang Dai, Qixiang Ye, Mingliang Xu, Yongjian Wu, Rongrong JiarXiv 2023 · DOI · arXiv

Abstract

Visible-Infrared Person Re-Identification (VI-ReID) is a critical task for 24/7 intelligent surveillance, yet it is hindered by the significant "modality gap" between visible (VIS) and near-infrared (NIR) spectra. Standard training losses, such as Softmax, often fail to explicitly account for these discrepancies, leading to suboptimal feature alignment. This paper introduces Spectral-Aware Softmax (SA-Softmax), a novel loss function designed to force cross-spectral alignment. By optimizing samples against prototypes from the opposite spectrum and employing a feature masking strategy, SA-Softmax significantly boosts retrieval accuracy in low-light scenarios, achieving state-of-the-art results on major benchmarks.

Addressing the Modality Gap

VI-ReID systems must match pedestrians across different camera types: visible cameras during the day and NIR cameras at night. The fundamental challenge is that the same person appears differently across these spectra due to varying wavelengths and lighting conditions.

Existing approaches often rely on standard Softmax loss, which treats VIS and NIR images of the same identity as a single class. However, Softmax is inherently modality-blind; it does not provide an explicit penalty for the distance between VIS and NIR features of the same person. As a result, a query image may be closer to a different person in the same spectrum (intra-modality inter-class) than to the correct person in a different spectrum (cross-modality intra-class).

Figure 2: Comparison of feature distributions between standard Softmax (a), where the modality gap $\theta_1$ exceeds the inter-class margin $\theta_2$, and SA-Softmax (b), which successfully aligns cross-modal features.

The SA-Softmax Framework

The core of the proposed method is the Spectral-Aware Softmax (SA-Softmax), which rethinks the classification process to bridge the spectral divide. The framework consists of three primary innovations:

  1. Asynchronous Modality Prototypes: Unlike standard methods that use a single prototype per identity, SA-Softmax maintains two distinct prototypes: $W^v$ for visible and $W^n$ for infrared. During training, a VIS sample is optimized to align with its corresponding NIR prototype, and vice versa. This "cross-spectral pulling" forces the two distributions into a shared latent space.
  2. Feature Mask (FM): To prevent the model from simply clustering same-modality features, the Feature Mask excludes the same-modality prototype from the loss calculation. This ensures the optimization gradient is entirely focused on bridging the spectral gap.
  3. Absolute-Similarity Term (AST): While Softmax is a relative loss, AST adds an explicit penalty to maximize the absolute cosine similarity between a sample and its target cross-modal prototype, leading to tighter, more compact identity clusters.
Figure 6: Overview of the SA-Softmax training process, illustrating the backbone network feeding into a prototype matrix composed of both visible and infrared representatives.

Experimental Results

The authors evaluated SA-Softmax on two primary VI-ReID benchmarks, demonstrating its effectiveness as a "plug-and-play" enhancement for standard backbones like ResNet-50.

  • RegDB: The method achieved a 95.2% Rank-1 accuracy and 88.7% mAP in Visible-to-Thermal search, significantly outperforming previous state-of-the-art (SOTA) models like CAJ and MCLNet.
  • SYSU-MM01: In the challenging "All-Search" mode, SA-Softmax reached 71.0% Rank-1 accuracy. While the improvement here was slightly less pronounced than on RegDB due to the dataset's complex pose and background variations, it still established a new performance baseline.

Visualization via t-SNE plots confirms that while standard Softmax leaves VIS and NIR features in separate clusters, SA-Softmax successfully merges them into unified, compact identity representations.

Conclusion

SA-Softmax addresses the modality bias inherent in standard classification losses for VI-ReID. By utilizing asynchronous prototypes and cross-modality optimization, the framework effectively bridges the spectral gap. Future work may involve integrating SA-Softmax with advanced data augmentation or adversarial training to further mitigate the impact of extreme pose and viewpoint variations in complex environments.

Summary updated

Abstract

Visible-infrared person re-identification (VI-ReID) aims to match specific pedestrian images from different modalities. Although suffering an extra modality discrepancy, existing methods still follow the softmax loss training paradigm, which is widely used in single-modality classification tasks. The softmax loss lacks an explicit penalty for the apparent modality gap, which adversely limits the performance upper bound of the VI-ReID task. In this paper, we propose the spectral-aware softmax (SA-Softmax) loss, which can fully explore the embedding space with the modality information and has clear interpretability. Specifically, SA-Softmax loss utilizes an asynchronous optimization strategy based on the modality prototype instead of the synchronous optimization based on the identity prototype in the original softmax loss. To encourage a high overlapping between two modalities, SA-Softmax optimizes each sample by the prototype from another spectrum. Based on the observation and analysis of SA-Softmax, we modify the SA-Softmax with the Feature Mask and Absolute-Similarity Term to alleviate the ambiguous optimization during model training. Extensive experimental evaluations conducted on RegDB and SYSU-MM01 demonstrate the superior performance of the SA-Softmax over the state-of-the-art methods in such a cross-modality condition.
BibTeX
@misc{tan2023spectral,
  author = {Lei Tan and Pingyang Dai and Qixiang Ye and Mingliang Xu and Yongjian Wu and Rongrong Ji},
  title = {{Spectral Aware Softmax for Visible-Infrared Person Re-Identification}},
  year = {2023},
  eprint = {2302.01512},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2302.01512},
}
Figure