Richer Semantics, Better Alignment: Aligning Visual Features with Explicit and Enriched Semantics for Visible-Infrared Person Re-Identification

Yimin Jiang, Huibing Wang, Jinjia Peng, Xianping Fu2024 · DOI

Abstract

Person search is a complex computer vision task that integrates pedestrian detection and re-identification. A primary challenge in this field is "scene noise"—environmental interference such as background clutter, lighting variations, and weather conditions that contaminate person-specific features. We present the Scene-Adaptive Person Search (SEAS) framework, which utilizes a bilateral modulation strategy to decouple identity features from environmental noise. By independently filtering background residuals and correcting foreground distortions, SEAS extracts a "pure" person representation that remains consistent across diverse camera views. Our approach achieves state-of-the-art performance on the CUHK-SYSU and PRW benchmarks, demonstrating significant improvements in cross-camera robustness.

Introduction

The goal of person search is to identify a specific individual across a network of cameras. Unlike standard re-identification, which uses pre-cropped images, person search must handle raw, unconstrained scenes. The major hurdle is scene noise: when a model crops a person, the resulting feature embedding often includes background pixels (background noise) and environmental effects like shadows or glare (foreground noise). Existing models often "bind" the person’s identity to these scene-specific features, causing recognition to fail when the individual moves to a different environment. SEAS addresses this by treating the scene as a subtractive element rather than a learned context.

Methodology: Bilateral Modulation

SEAS introduces a bilateral modulation strategy consisting of two specialized networks designed to strip away noise and isolate the core identity feature.

The SEAS bilateral modulation process, decomposing a raw person feature into background noise, scene/foreground noise, and a final pure identity feature.

1. Background Modulation Network (BMN)

The BMN focuses on the spatial noise within the detected bounding box. It utilizes a Multi-Granularity Embedding (MGE) to capture both global and fine-grained horizontal details. To suppress non-person pixels, the authors implement a Background Noise Reduction (BNR) loss. This loss leverages the mathematical norm of feature vectors, supervising the model to distinguish between high-strength "person" features and low-strength "background" features without requiring expensive pixel-level masks.

2. Foreground Modulation Network (FMN)

The FMN addresses environmental noise (e.g., lighting, weather) by analyzing the context outside the bounding box. It extracts a global noise map from the entire scene and uses a Feature Denoiser—a modified Transformer-style decoder—to compare the person’s embedding with the scene's noise profile. The network calculates a "clutter reduction offset," which is mathematically subtracted from the person's feature to neutralize environmental interference.

3. Bidirectional Online Instance Matching (BOIM) Loss

To enhance discriminative power, SEAS employs the BOIM loss. This improves upon standard matching losses by incorporating a triplet component that maximizes the distance between different individuals in the feature space, ensuring that the "pure" features are not only clean but also highly unique.

Experimental Results

SEAS was evaluated on two gold-standard datasets, consistently outperforming existing "scene-binding" methods:

  • CUHK-SYSU: The model achieved 97.1% mAP and 97.8% top-1 accuracy, setting a new benchmark for large-scale variety.
  • PRW (Person Search in the Wild): SEAS reached 60.5% mAP, a significant lead over other end-to-end frameworks.
  • Cross-Camera Robustness: In specialized tests measuring performance drops between different camera views, SEAS exhibited superior stability, proving that its denoising process effectively decouples identity from the surrounding scene.

Conclusion

The SEAS framework demonstrates that treating environmental interference as bilateral noise—both spatial and contextual—is key to robust person search. By actively subtracting these components through the BMN and FMN, the model produces a pure identity representation that is highly resilient to scene changes. Future work will explore the application of bilateral modulation to real-time tracking and extreme occlusion scenarios.

Summary updated

Abstract

Visible-infrared person re-identification (VIReID) retrieves pedestrian images with the same identity across different modalities. Existing methods learn visual features solely from images, failing to align them into the modality-invariant semantic space. In this paper, we propose a novel framework, termed Richer Semantics, Better Alignment (RSBA), to align visual features with explicit and enriched semantics. Specifically, we first develop an Explicit Semantics-Guided Feature Alignment (ESFA) module, which supplements textual descriptions for cross-modality images and aligns image-text pairs within each modality, alleviating the distribution discrepancy of visual features. We then devise a Consistent Similarity-Guided Indirect Alignment (CSIA) module, which constrains the similarity between intra-modality image-text pairs to be consistent with that between inter-modality text-text pairs, indirectly aligning visual features with cross-modality semantics. Furthermore, we design a Cross-View Semantics Compensation (CVSC) module, which integrates multi-view texts and improves the image-text matching of one-to-one in ESFA and CSIA to one-to-many, further strengthening the alignment of visual features within the semantic space. Extensive experimental results on three public datasets demonstrate the effectiveness and superiority of our proposed RSBA.
BibTeX
@misc{jiang2024richer,
  author = {Yimin Jiang and Huibing Wang and Jinjia Peng and Xianping Fu},
  title = {{Richer Semantics, Better Alignment: Aligning Visual Features with Explicit and Enriched Semantics for Visible-Infrared Person Re-Identification}},
  year = {2024},
  doi = {10.24963/ijcai.2024/104},
}
Figure