Scene-Adaptive Person Search via Bilateral Modulations

Yimin Jiang, Huibing Wang, Jinjia Peng, Xianping Fu, Yang Wang

Abstract

Person search involves localizing a target individual within a diverse gallery of images. A significant challenge in this task is "scene noise"—background and foreground elements that corrupt the person's identity features as they move through different environments. We present Scene-Adaptive Person Search (SEAS), a model that utilizes bilateral modulations to eliminate scene noise and maintain consistent person representations. SEAS introduces a Background Modulation Network (BMN) to reduce background noise within detected bounding boxes and a Foreground Modulation Network (FMN) to mitigate environmental effects like lighting. Our approach achieves state-of-the-art performance on the CUHK-SYSU (97.1% mAP) and PRW (60.5% mAP) benchmarks.

Introduction: The Challenge of Scene Noise

In person search, the goal is to detect a person and then correctly identify them (re-identification) across different camera views. However, a person's captured appearance is rarely "pure." When a bounding box is generated, it often includes background noise (e.g., street elements, vehicles) and is affected by foreground noise (e.g., shadows, lighting changes).

The SEAS framework aims to decompose a noisy person feature into its constituent noise components to extract a "Pure Feature" for robust re-identification.

Existing methods often either ignore these environmental factors or "bind" the scene to the person's identity. The latter is problematic because if the person moves to a new location, the previously learned scene information becomes misleading. SEAS shifts this paradigm by using scene information as a filter to clean the person's features rather than as an additive component.

Proposed Method: Scene-Adaptive Person Search (SEAS)

The SEAS framework employs a bilateral modulation strategy to isolate and remove noise at two levels:

1. Background Modulation Network (BMN)

The BMN focuses on noise within the detected bounding box. It utilizes a Multi-Granularity Embedding (MGE) to partition the person's image into horizontal strips, preserving fine-grained local details. To suppress non-person pixels, it implements a Background Noise Reduction (BNR) loss. This loss is "norm-aware," using the L2-norm of features to distinguish between high-information identity features and low-information background clutter without requiring expensive pixel-level segmentation.

2. Foreground Modulation Network (FMN)

The FMN addresses broader environmental noise. By extracting a feature map of the entire scene and applying a cross-attention mechanism, the FMN calculates a "clutter reduction offset." This offset is added to the person's embedding to effectively "cancel out" the impact of varying lighting or weather conditions, ensuring the identity representation remains stable across different scenes.

3. Bidirectional Online Instance Matching (BOIM) Loss

To enhance the re-identification sub-task, SEAS uses the BOIM loss, which combines standard Online Instance Matching (OIM) with a triplet loss. This dual approach helps the model better distinguish between similar-looking individuals (inter-person differences) while recognizing the same person across different poses and viewpoints (intra-person similarity).

Experimental Results

SEAS was evaluated on two primary benchmarks: CUHK-SYSU and PRW (Person Search in the Wild).

Dataset mAP Top-1 Accuracy
CUHK-SYSU 97.1% 97.8%
PRW 60.5% -

These results represent new state-of-the-art (SOTA) benchmarks. Notably, SEAS demonstrated superior robustness in cross-camera scenarios, confirming that its "noise cancellation" strategy effectively decouples identity from environmental context.

Conclusion

By treating scene information as a source of noise to be modulated out rather than features to be integrated, SEAS provides a more robust and adaptive solution for person search. While future work may address computational overhead and extreme occlusions, the bilateral modulation approach sets a new standard for maintaining consistent identity representations in dynamic environments.

The code is available at: https://github.com/whbdmu/SEAS.

Summary updated

Abstract

Person search aims to localize specific a target person from a gallery set of images with various scenes. As the scene of moving pedestrian changes, the captured person image inevitably bring in lots of background noise and foreground noise on the person feature, which are completely unrelated to the person identity, leading to severe performance degeneration. To address this issue, we present a Scene-Adaptive Person Search (SEAS) model by introducing bilateral modulations to simultaneously eliminate scene noise and maintain a consistent person representation to adapt to various scenes. In SEAS, a Background Modulation Network (BMN) is designed to encode the feature extracted from the detected bounding box into a multi-granularity embedding, which reduces the input of background noise from multiple levels with norm-aware. Additionally, to mitigate the effect of foreground noise on the person feature, SEAS introduces a Foreground Modulation Network (FMN) to compute the clutter reduction offset for the person embedding based on the feature map of the scene image. By bilateral modulations on both background and foreground within an end-to-end manner, SEAS obtains consistent feature representations without scene noise. SEAS can achieve state-of-the-art (SOTA) performance on two benchmark datasets, CUHK-SYSU with 97.1% mAP and PRW with 60.5% mAP. The code is available at https://github.com/whbdmu/SEAS.
BibTeX
@inproceedings{jiang2024scene,
  author = {Yimin Jiang and Huibing Wang and Jinjia Peng and Xianping Fu and Yang Wang},
  title = {{Scene-Adaptive Person Search via Bilateral Modulations}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
  url = {https://openreview.net/forum?id=GjxjiSapmk},
}
Figure