Abstract
Occluded person re-identification (re-ID) poses a significant challenge due to partial visibility and spatial misalignment. Traditional methods rely on external cues like pose estimation or human parsing to align body parts, which are often fragile and computationally expensive. This paper introduces Matching on Sets (MoS), a novel framework that treats person images as sets of visual patterns. By employing a differentiable Jaccard similarity metric and a pattern conflict penalty, MoS achieves state-of-the-art performance on occluded datasets without requiring spatial alignment or external models.
Introduction
Person re-identification (re-ID) aims to identify the same individual across non-overlapping camera views. While holistic re-ID has matured, occluded re-ID remains difficult because obstacles like cars or trees hide parts of the body. Most current solutions attempt to solve this via spatial alignment—using auxiliary models to locate body parts. However, these methods are often computationally heavy and fail if the pose estimator itself is hindered by occlusion.
MoS addresses these issues by abandoning spatial alignment entirely. Instead of asking "where" a body part is, it asks "which" visual patterns are present, treating the image as a collection of semantic features.
Methodology
The MoS framework shifts the re-ID task from spatial matching to set matching.
Pattern Set Representation
The core idea is to interpret the output channels of a Convolutional Neural Network (CNN) as a "pattern set." Each channel is hypothesized to respond to specific visual concepts, such as a particular texture, color, or accessory. A person is thus represented by a global vector where each element signifies the presence of a specific pattern.
Differentiable Jaccard Similarity
To compare two pattern sets, MoS utilizes the Jaccard Similarity (Intersection over Union). Because standard Jaccard logic applies to binary sets, the authors adapt it for continuous neural network outputs using minimization and maximization to approximate "AND" and "OR" operations. A "Softmax-Jaccard" version is introduced to ensure the metric is fully differentiable during training.
Jaccard Triplet Loss
During training, the model is optimized using a specialized triplet loss based on the Jaccard Distance. This ensures that images of the same identity have highly overlapping pattern sets (high intersection), while images of different identities are pushed apart in the pattern space. This approach aligns the training objective directly with the set-matching inference logic.
Pattern Conflict Penalty
A key innovation in MoS is the Pattern Conflict Penalty, applied during the inference stage. This mechanism incorporates "common sense" by penalizing similarity scores if the combined pattern set contains mutually exclusive features. For example, a single person is unlikely to wear both blue and black shoes simultaneously.
Experimental Results
The authors evaluated MoS on several benchmarks, demonstrating its superiority in handling occlusions:
- Occluded-DukeMTMC: MoS achieved 61.0% Rank-1 accuracy, significantly outperforming previous state-of-the-art methods like PGFA (51.4%) and HOnet (55.1%).
- P-ETHZ: The model reached 76.2% Rank-1, an 18% improvement over existing alignment-based methods.
- Efficiency: By removing the need for separate pose estimation or parsing branches, MoS is faster and simpler to deploy than its competitors.
- Holistic Datasets: MoS maintained competitive performance on standard datasets like Market-1501, proving its robustness even when occlusions are absent.
Conclusion
Matching on Sets (MoS) provides a robust and efficient alternative to alignment-based person re-ID. By treating images as sets of semantic patterns and leveraging the Jaccard metric, it avoids the pitfalls of fragile pose estimation. The inclusion of a pattern conflict penalty further enhances its discriminative power, setting a new benchmark for occluded person re-identification.
Summary updated