Matching on Sets: Conquer Occluded Person Re-identification Without Alignment

Mengxi Jia, Xinhua Cheng, Yunpeng Zhai, Shijian Lu, Siwei Ma, Yonghong Tian, Jian Zhang

Abstract

Occluded person re-identification (re-ID) poses a significant challenge due to partial visibility and spatial misalignment. Traditional methods rely on external cues like pose estimation or human parsing to align body parts, which are often fragile and computationally expensive. This paper introduces Matching on Sets (MoS), a novel framework that treats person images as sets of visual patterns. By employing a differentiable Jaccard similarity metric and a pattern conflict penalty, MoS achieves state-of-the-art performance on occluded datasets without requiring spatial alignment or external models.

Introduction

Person re-identification (re-ID) aims to identify the same individual across non-overlapping camera views. While holistic re-ID has matured, occluded re-ID remains difficult because obstacles like cars or trees hide parts of the body. Most current solutions attempt to solve this via spatial alignment—using auxiliary models to locate body parts. However, these methods are often computationally heavy and fail if the pose estimator itself is hindered by occlusion.

MoS addresses these issues by abandoning spatial alignment entirely. Instead of asking "where" a body part is, it asks "which" visual patterns are present, treating the image as a collection of semantic features.

Methodology

The MoS framework shifts the re-ID task from spatial matching to set matching.

Pattern Set Representation

The core idea is to interpret the output channels of a Convolutional Neural Network (CNN) as a "pattern set." Each channel is hypothesized to respond to specific visual concepts, such as a particular texture, color, or accessory. A person is thus represented by a global vector where each element signifies the presence of a specific pattern.

The MoS architecture: Images are processed through a backbone to extract pattern sets, which are then compared using Jaccard similarity and a conflict penalty.

Differentiable Jaccard Similarity

To compare two pattern sets, MoS utilizes the Jaccard Similarity (Intersection over Union). Because standard Jaccard logic applies to binary sets, the authors adapt it for continuous neural network outputs using minimization and maximization to approximate "AND" and "OR" operations. A "Softmax-Jaccard" version is introduced to ensure the metric is fully differentiable during training.

Jaccard Triplet Loss

During training, the model is optimized using a specialized triplet loss based on the Jaccard Distance. This ensures that images of the same identity have highly overlapping pattern sets (high intersection), while images of different identities are pushed apart in the pattern space. This approach aligns the training objective directly with the set-matching inference logic.

Pattern Conflict Penalty

A key innovation in MoS is the Pattern Conflict Penalty, applied during the inference stage. This mechanism incorporates "common sense" by penalizing similarity scores if the combined pattern set contains mutually exclusive features. For example, a single person is unlikely to wear both blue and black shoes simultaneously.

The Pattern Conflict Penalty identifies incompatible visual patterns within the union of two sets to refine similarity scores.

Experimental Results

The authors evaluated MoS on several benchmarks, demonstrating its superiority in handling occlusions:

  • Occluded-DukeMTMC: MoS achieved 61.0% Rank-1 accuracy, significantly outperforming previous state-of-the-art methods like PGFA (51.4%) and HOnet (55.1%).
  • P-ETHZ: The model reached 76.2% Rank-1, an 18% improvement over existing alignment-based methods.
  • Efficiency: By removing the need for separate pose estimation or parsing branches, MoS is faster and simpler to deploy than its competitors.
  • Holistic Datasets: MoS maintained competitive performance on standard datasets like Market-1501, proving its robustness even when occlusions are absent.

Conclusion

Matching on Sets (MoS) provides a robust and efficient alternative to alignment-based person re-ID. By treating images as sets of semantic patterns and leveraging the Jaccard metric, it avoids the pitfalls of fragile pose estimation. The inclusion of a pattern conflict penalty further enhances its discriminative power, setting a new benchmark for occluded person re-identification.

Summary updated

Abstract

Occluded person re-identification (re-ID) is a challenging task as different human parts may become invisible in cluttered scenes, making it hard to match person images of different identities. Most existing methods address this challenge by aligning spatial features of body parts according to semantic information (e.g. human poses) or feature similarities but this approach is complicated and sensitive to noises. This paper presents Matching on Sets (MoS), a novel method that positions occluded person re-ID as a set matching task without requiring spatial alignment. MoS encodes a person image by a pattern set as represented by a `global vector’ with each element capturing one specific visual pattern, and it introduces Jaccard distance as a metric to compute the distance between pattern sets and measure image similarity. To enable Jaccard distance over continuous real numbers, we employ minimization and maximization to approximate the operations of intersection and union, respectively. In addition, we design a Jaccard triplet loss that enhances the pattern discrimination and allows to embed set matching into deep neural networks for end-to-end training. In the inference stage, we introduce a conflict penalty mechanism that detects mutually exclusive patterns in the pattern union of image pairs and decreases their similarities accordingly. Extensive experiments over three widely used datasets (Market1501, DukeMTMC and Occluded-DukeMTMC) show that MoS achieves superior re-ID performance. Additionally, it is tolerant of occlusions and outperforms the state-of-the-art by large margins for Occluded-DukeMTMC.
BibTeX
@inproceedings{jia2021matching,
  author = {Mengxi Jia and Xinhua Cheng and Yunpeng Zhai and Shijian Lu and Siwei Ma and Yonghong Tian and Jian Zhang},
  title = {{Matching on Sets: Conquer Occluded Person Re-identification Without Alignment}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.1609/aaai.v35i2.16260},
}
Figure