Region Generation and Assessment Network for Occluded Person Re-Identification

Shuting He, Weihua Chen, Kai Wang, Hao Luo, Fan Wang, Wei Jiang, Henghui Ding

TL;DR

RGANet is a novel framework for person re-identification (ReID) that tackles the challenges of occlusion and misalignment by leveraging the semantic power of the CLIP vision-language model. It adaptively generates human body regions using learnable text prompts and filters out noisy or occluded parts using a dual-indicator assessment module, achieving state-of-the-art performance with high computational efficiency.

What Problem Does This Paper Address?

Person Re-identification (ReID) is the task of matching a specific individual across different camera views and times. In real-world surveillance, this is rarely straightforward; people are often partially hidden by cars, trees, or other pedestrians (occlusion) or appear in different parts of the frame (misalignment).

Current solutions generally fall into two camps:

  1. External Cues: Using off-the-shelf models (like pose estimators or semantic parsers) to find body parts. However, these models often suffer from a "domain gap"—they weren't trained on ReID data and may fail in grainy or crowded surveillance footage.
  2. Matching Strategies: Comparing local image patches. While accurate, these are often computationally expensive and slow during inference.

The authors of RGANet aim to bridge this gap by creating a system that "understands" body parts semantically without needing heavy external tools or slow matching algorithms.

Methods

The RGANet architecture consists of two primary components:

1. Region Generation Module (RGM)

Instead of using a separate pose estimator, the RGM utilizes a pre-trained CLIP (Contrastive Language-Image Pre-training) model.

  • Semantic Prototypes: The authors feed text descriptions like "head," "upper body," "lower body," and "foot" into CLIP’s text encoder to generate feature vectors (prototypes).
  • Learnable Prompts: To solve the domain gap, they introduce "learnable context tokens." Instead of a static prompt like "a photo of a [CLASS]," the model learns the optimal word embeddings to describe body parts specifically within the context of ReID datasets.
  • Adaptive Segmentation: By calculating the cosine similarity between the image pixels and these textual prototypes, the model generates a segmentation mask to isolate specific body regions.

2. Region Assessment Module (RAM)

Not all detected regions are useful. If a person's legs are hidden by a car, that "lower body" region contains noise. The RAM assigns a confidence score to each region based on two indicators:

  • Discrimination-Aware Indicator: A self-learning module that measures how well a specific region helps distinguish between different identities.
  • Invariance-Aware Indicator: This uses a memory bank to store "ideal" features for each body part. It compares a detected region (e.g., a head) to the global average of "heads" in the dataset. If the region looks nothing like a typical head, it receives a lower score.

Datasets

The authors evaluated RGANet across six widely-used benchmarks covering three distinct scenarios:

  • Occluded ReID: Occluded-DukeMTMC and Occluded-REID.
  • Partial ReID: Partial-REID and Partial-iLIDS.
  • Holistic ReID: Market-1501 and MSMT17 (large-scale).

Key Insights and Ideas

  • CLIP as a Zero-Shot Part Detector: The most elegant idea here is using CLIP's cross-modal alignment to perform part detection. By treating part localization as a "text-to-pixel" matching problem, the model gains a semantic understanding that is more robust than simple geometric partitioning.
  • Bypassing External Tools: Because the textual prototypes are optimized during training, they can be used directly during inference. This removes the need to run a separate, heavy pose-estimation model, making the system much faster.
  • The Power of Invariance: The use of a memory bank to enforce "invariance" is a clever way to detect occlusion. It essentially asks: "Does this part look like what a human [head/arm/leg] is supposed to look like?" If the answer is no, the model automatically down-weights that feature.

Key Results

RGANet consistently outperformed previous state-of-the-art (SOTA) methods:

  • Accuracy: On the challenging Occluded-DukeMTMC dataset, it achieved 71.6% Rank-1 accuracy and 62.4% mAP, surpassing previous leaders like DPM and FED.
  • Generalization: On the large-scale MSMT17 dataset, it reached 88.1% Rank-1, a significant jump (+1.9%) over the previous SOTA (TransReID).
  • Efficiency: RGANet is significantly faster than external-tool-based methods. It runs at 213.9 FPS, compared to HOReID’s 62.9 FPS, making it much more viable for real-time applications.

Limitations and Future Work

  • Hyper-parameter Sensitivity: The performance is somewhat sensitive to the "amplification factor" ($\gamma$) used in the similarity calculation and the length of the learnable prompts ($K$).
  • Static Images: The current work focuses on single-frame images. Extending this semantic region assessment to video-based ReID, where temporal consistency could further help resolve occlusions, is a natural next step.
  • Background Noise: While the "background" class helps, extremely cluttered environments where the background looks like clothing might still pose a challenge for the RGM.

Summary updated

Abstract

Person Re-identification (ReID) plays a more and more crucial role in recent years with a wide range of applications. Existing ReID methods are suffering from the challenges of misalignment and occlusions, which degrade the performance dramatically. Most methods tackle such challenges by utilizing external tools to locate body parts or exploiting matching strategies. Nevertheless, the inevitable domain gap between the datasets utilized for external tools and the ReID datasets and the complicated matching process make these methods unreliable and sensitive to noises. In this paper, we propose a Region Generation and Assessment Network (RGANet) to effectively and efficiently detect the human body regions and highlight the important regions. In the proposed RGANet, we first devise a Region Generation Module (RGM) which utilizes the pre-trained CLIP to locate the human body regions using semantic prototypes extracted from text descriptions. Learnable prompt is designed to eliminate domain gap between CLIP datasets and ReID datasets. Then, to measure the importance of each generated region, we introduce a Region Assessment Module (RAM) that assigns confidence scores to different regions and reduces the negative impact of the occlusion regions by lower scores. The RAM consists of a discrimination-aware indicator and an invariance-aware indicator, where the former indicates the capability to distinguish from different identities and the latter represents consistency among the images of the same class of human body regions. Extensive experimental results for six widely-used benchmarks including three tasks (occluded, partial, and holistic) demonstrate the superiority of RGANet against state-of-the-art methods.
BibTeX
@article{he2024region,
  author = {Shuting He and Weihua Chen and Kai Wang and Hao Luo and Fan Wang and Wei Jiang and Henghui Ding},
  title = {{Region Generation and Assessment Network for Occluded Person Re-Identification}},
  journal = {IEEE Trans. Inf. Forensics Secur.},
  year = {2024},
  url = {https://openreview.net/forum?id=2qCXtCIkIQ},
}

Loading related content...

Figure