Pose Guided Gated Fusion for Person Re-identification

Amran Bhuiyan, Yang Liu, Parthipan Siva, Mehrsan Javan, Ismail Ben Ayed, Eric Granger

Introduction

Person re-identification (Re-ID) is a critical task in computer vision, aimed at identifying the same individual across multiple non-overlapping camera views. Despite significant progress, Re-ID remains challenging due to dramatic variations in human pose and frequent spatial misalignment caused by imperfect person detection or occlusions. Traditional models often focus on global appearance features, which can be unreliable when a person's orientation changes or when they are partially visible.

Proposed Methodology

To address these issues, this research introduces a novel architecture that leverages human pose information to dynamically modulate appearance features. The system consists of three primary components:

  1. Backbone Network: A standard convolutional neural network (CNN), such as ResNet-50, which extracts fundamental appearance features from the input image.
  2. Pose Network: Utilizing a pre-trained model (e.g., OpenPose), this component generates pose maps (heatmaps) identifying the spatial locations of key joints and limbs.
  3. Gated Fusion Network: This module serves as the core innovation, acting as a bridge between the backbone and pose networks.
Figure 1: The proposed architecture featuring the Backbone Net, Pose Net, and the novel Gated Fusion Net for mid-level feature integration.

Unlike traditional "late fusion" methods that combine information only at the final layers, the Gated Fusion module integrates pose data at the mid-level layers (specifically C3 and C4). By employing a spatial and channel-wise attention mechanism, the pose network "gates" the backbone's features, directing the model to focus on relevant body parts while suppressing background noise and irrelevant pixels.

Key Insights

A pivotal finding of this work is the effectiveness of mid-level fusion. Fusing at early layers introduces too much noise, while late fusion is often too abstract to capture the fine-grained relationship between pose and appearance. Mid-level layers represent "parts" and "textures" that align well with structural pose information. This contextual gating mechanism allows the network to remain robust even when the initial bounding box is poorly cropped or the person is off-center.

Experimental Evaluation

The model was evaluated on three major Re-ID benchmarks: Market-1501, CUHK03-NP, and DukeMTMC-reID.

  • Market-1501: When integrated with a strong baseline (Bag of Tricks), the model achieved a Rank-1 accuracy of 94.60% and an mAP of 87.10%.
  • CUHK03-NP: This dataset, characterized by significant misalignment, saw the most substantial gains. The method provided a 7.42% improvement in Rank-1 accuracy over the baseline on detected images, demonstrating its superior handling of spatial variations.

Qualitative analysis further supports these results. While standard models may struggle to distinguish between individuals wearing similar clothing, the pose-guided approach successfully differentiates them by focusing on body shape and clothing folds specific to the detected pose.

Figure 5: Example of a challenging person image where pose-guided features help resolve ambiguity.

Conclusion and Future Work

The Pose Guided Gated Fusion network significantly enhances Re-ID performance by treating pose as a contextual selector for appearance features. Future research will explore end-to-end training of the entire pipeline and methods to reduce the computational overhead associated with running a full pose estimation network in real-time scenarios.

Summary updated

Abstract

Person re-identification is an important yet challenging problem in visual recognition. Despite the recent advances with deep learning (DL) models for spatio-temporal and multi-modal fusion, re-identification approaches often fail to leverage the contextual information (e.g., pose and illumination) to dynamically select the most discriminant con-volutional filters (i.e., appearance features) for feature representation and inference. State-of-the-art techniques for gated fusion employ complex dedicated part- or attention-based architectures for late fusion, and do not incorporate pose and appearance information to train the backbone network. In this paper, a new DL model is proposed for pose-guided re-identification, comprised of a deep backbone, pose estimation, and gated fusion network. Given a query image of an individual, the backbone convolutional NN produces a feature embedding required for pair-wise matching with embeddings for reference images, where feature maps from the pose network and from mid-level CNN layers are combined by the gated fusion network to generate pose-guided gating. The proposed framework allows to dynamically activate the most discriminant CNN filters based on pose information in order to perform a finer grained recognition. Extensive experiments on three challenging benchmark datasets indicate that integrating the pose-guided gated fusion into the state-of-the-art re-identification backbone architecture allows to improve their recognition accuracy. Experimental results also support our intuition on the advantages of gating backbone appearance information using the pose feature maps at mid-level CNN layers.
BibTeX
@inproceedings{bhuiyan2020pose,
  author = {Amran Bhuiyan and Yang Liu and Parthipan Siva and Mehrsan Javan and Ismail Ben Ayed and Eric Granger},
  title = {{Pose Guided Gated Fusion for Person Re-identification}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2020},
  doi = {10.1109/wacv45572.2020.9093370},
}
Figure