Region Generation and Assessment Network for Occluded Person Re-Identification

Shuting He, Weihua Chen, Kai Wang, Hao Luo, Fan Wang, Wei Jiang, Henghui Ding

Abstract

Person Re-identification (ReID) plays an increasingly crucial role in modern surveillance, yet existing methods struggle with misalignment and occlusions. Most current approaches rely on external tools for body part localization or complex matching strategies, which are often unreliable due to domain gaps and high computational costs. This paper proposes the Region Generation and Assessment Network (RGANet) to effectively and efficiently detect and evaluate human body regions. RGANet introduces a Region Generation Module (RGM) that utilizes pre-trained CLIP and learnable prompts to locate body parts semantically. Additionally, a Region Assessment Module (RAM) assigns confidence scores to these regions using discrimination-aware and invariance-aware indicators, mitigating the impact of occlusions. Extensive experiments across six benchmarks demonstrate RGANet's superiority in occluded, partial, and holistic ReID tasks, achieving state-of-the-art performance with high efficiency.

Introduction: The Challenge of Occlusion and Misalignment

In real-world surveillance, matching a specific individual across different camera views is complicated by environmental factors. Pedestrians are frequently partially hidden by cars, trees, or other people (occlusion) or appear in different spatial locations within the frame (misalignment). These challenges degrade ReID performance by introducing noise and missing critical identity features.

The challenge of occluded person ReID: matching a partially hidden query image to a full-body gallery image.

Traditional solutions generally fall into two categories:

  1. External Cues: Using off-the-shelf models like pose estimators to find body parts. However, these often suffer from a domain gap between their training data and surveillance footage.
  2. Matching Strategies: Comparing local image patches or using graph-based representations. While accurate, these are often computationally expensive and slow during inference.
Common matching strategies in ReID, including graph-based, part-based, and set-based approaches.

The RGANet Framework

RGANet aims to bridge the gap between semantic understanding and computational efficiency through two primary components.

Region Generation Module (RGM)

Instead of relying on heavy external pose estimators, the RGM utilizes the pre-trained CLIP (Contrastive Language-Image Pre-training) model. It generates semantic prototypes from text descriptions like "head" or "upper body." To overcome the domain gap, the module uses learnable prompts, allowing the model to optimize word embeddings specifically for ReID datasets. By calculating the cosine similarity between image pixels and these textual prototypes, the model adaptively segments specific body regions.

Region Assessment Module (RAM)

To filter out noisy or occluded regions, the RAM assigns a confidence score to each part based on two indicators:

  • Discrimination-Aware Indicator: Measures how well a specific region helps distinguish between different identities.
  • Invariance-Aware Indicator: Uses a memory bank to compare a detected region against the dataset's "ideal" features for that body part. If a region (e.g., a leg) is occluded by a car, it will look nothing like a typical leg, and the RAM will automatically down-weight its importance.

Key Results and Efficiency

RGANet was evaluated on six widely-used benchmarks, consistently outperforming previous state-of-the-art (SOTA) methods:

  • Accuracy: On the challenging Occluded-DukeMTMC dataset, it achieved 71.6% Rank-1 accuracy and 62.4% mAP, surpassing previous leaders like DPM and FED.
  • Generalization: On the large-scale MSMT17 dataset, it reached 88.1% Rank-1, a 1.9% improvement over the previous SOTA, TransReID.
  • Efficiency: By bypassing external tools, RGANet runs at 213.9 FPS, making it significantly faster than methods like HOReID (62.9 FPS) and more viable for real-time applications.

Conclusion and Future Work

RGANet demonstrates that leveraging the semantic power of vision-language models like CLIP can effectively solve the problems of occlusion and misalignment in ReID without the overhead of external tools. While the current model is sensitive to certain hyper-parameters and background noise, future work could extend this semantic assessment to video-based ReID, where temporal consistency could further enhance robustness.

Summary updated

Abstract

Person Re-identification (ReID) plays a more and more crucial role in recent years with a wide range of applications. Existing ReID methods are suffering from the challenges of misalignment and occlusions, which degrade the performance dramatically. Most methods tackle such challenges by utilizing external tools to locate body parts or exploiting matching strategies. Nevertheless, the inevitable domain gap between the datasets utilized for external tools and the ReID datasets and the complicated matching process make these methods unreliable and sensitive to noises. In this paper, we propose a Region Generation and Assessment Network (RGANet) to effectively and efficiently detect the human body regions and highlight the important regions. In the proposed RGANet, we first devise a Region Generation Module (RGM) which utilizes the pre-trained CLIP to locate the human body regions using semantic prototypes extracted from text descriptions. Learnable prompt is designed to eliminate domain gap between CLIP datasets and ReID datasets. Then, to measure the importance of each generated region, we introduce a Region Assessment Module (RAM) that assigns confidence scores to different regions and reduces the negative impact of the occlusion regions by lower scores. The RAM consists of a discrimination-aware indicator and an invariance-aware indicator, where the former indicates the capability to distinguish from different identities and the latter represents consistency among the images of the same class of human body regions. Extensive experimental results for six widely-used benchmarks including three tasks (occluded, partial, and holistic) demonstrate the superiority of RGANet against state-of-the-art methods.
BibTeX
@article{he2024region,
  author = {Shuting He and Weihua Chen and Kai Wang and Hao Luo and Fan Wang and Wei Jiang and Henghui Ding},
  title = {{Region Generation and Assessment Network for Occluded Person Re-Identification}},
  journal = {IEEE Trans. Inf. Forensics Secur.},
  year = {2024},
  url = {https://openreview.net/forum?id=2qCXtCIkIQ},
}
Figure