Robust Ensemble Person Re-Identification via Orthogonal Fusion with Occlusion Handling

Syeda Nyma Ferdous, Xin Li

Abstract

Person Re-Identification (ReID) remains a critical challenge in surveillance and autonomous systems, particularly when subjects are partially obscured by environmental obstacles. This paper introduces OFOH (Orthogonal Fusion with Occlusion Handling), a deep ensemble framework designed to mitigate the impact of occlusion. By integrating the local-context strengths of Convolutional Neural Networks (CNNs) with the global-reasoning capabilities of Transformers, and leveraging features from Masked Autoencoders (MAE), OFOH achieves state-of-the-art performance across multiple challenging benchmarks.

Introduction

In real-world scenarios, person ReID is often hindered by occlusion—where trees, vehicles, or other pedestrians block a subject's view. Standard ReID models typically fail in these conditions because occlusion introduces noise while simultaneously removing vital identifying signals. Previous solutions often relied on manual labeling or rigid masking, which lack the flexibility to handle irregular, real-world obstacles. OFOH addresses this by learning robust, occlusion-invariant features automatically through an ensemble of specialized experts.

Methodology

The OFOH framework is built upon two primary "experts" that provide complementary perspectives on the input data:

  1. DEM1 (The CNN Expert): This branch utilizes a ResNet-50 backbone enhanced with features from a Masked Autoencoder (MAE). Because MAEs are pre-trained to reconstruct missing image segments, their feature space is naturally adept at handling missing data. These features are integrated using Orthogonal Fusion, a technique that projects local and global features into an orthogonal space. This ensures that critical local details are preserved while redundant information is filtered out.
  2. DEM2 (The Transformer Expert): Based on the Vision Transformer (ViT) architecture, this branch focuses on "part-occluded tokens." It employs a CNN-based verifier to identify discriminative occluded samples, training the Transformer to remain robust even when specific image patches (tokens) are missing or shifted.

To further refine the model's focus, the authors utilize Sparsemax attention instead of standard Softmax. Sparsemax forces the model to assign zero attention to irrelevant, occluded pixels, effectively "muting" noise from obstacles.

Ensemble Strategies

The final identity prediction is generated by an ensemble layer that combines the outputs of DEM1 and DEM2. The framework supports two primary strategies for this fusion: Voting and Stacking.

The OFOH framework supports two ensemble strategies: Stacking (left), which uses an MLP to learn expert weights, and Voting (right), which averages prediction scores.

In the Voting approach, the model simply averages the scores from both experts. In the Stacking approach, a small Multi-Layer Perceptron (MLP) is trained to learn which expert to trust based on the input features. To prevent the experts from converging on identical solutions, diversity regularization is applied to keep their weight vectors distinct.

Experimental Results

OFOH was evaluated on several benchmarks, including specialized occluded datasets and standard holistic datasets:

  • Occluded-REID: The model achieved 81.9% Rank-1 accuracy and 83.1% mAP, significantly outperforming previous specialized models.
  • Occluded Duke: OFOH reached 71.7% Rank-1, demonstrating scalability on larger occluded datasets.
  • Market-1501: On standard, non-occluded data, the model maintained a highly competitive 95.6% Rank-1, proving that its occlusion-handling mechanisms do not degrade general performance.

Ablation studies confirmed the importance of the MAE-enhanced features; removing them resulted in a nearly 2% drop in accuracy, validating the hypothesis that reconstruction capabilities improve identification under occlusion.

Conclusion and Future Work

OFOH demonstrates that combining generative reconstruction with orthogonal feature fusion provides a powerful defense against occlusion in person ReID. However, challenges remain in aerial (UAV) surveillance, where extreme viewing angles and low resolution significantly reduce discriminative information. Future research will focus on multi-scale handling for varying distances and extending the framework to video-based ReID, where temporal clues can be leveraged to "see" through intermittent obstacles.

Summary updated

BibTeX
@misc{ferdous2024robust,
  author = {Syeda Nyma Ferdous and Xin Li},
  title = {{Robust Ensemble Person Re-Identification via Orthogonal Fusion with Occlusion Handling}},
  year = {2024},
  eprint = {2404.00107},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2404.00107},
}
Figure