GAN-based Pose-aware Regulation for Video-based Person Re-identification

Alessandro Borgia, Yang Hua, Elyor Kodirov, Neil Martin Robertson

Abstract

Video-based person re-identification deals with the inherent difficulty of matching unregulated sequences with different length and with incomplete target pose/viewpoint structure. Common approaches operate either by reducing the problem to the still images case, facing a significant information loss, or by exploiting inter-sequence temporal dependencies as in Siamese Recurrent Neural Networks or in gait analysis. However, in all cases, the inter-sequences pose/viewpoint misalignment is not considered, and the existing spatial approaches are mostly limited to the still images context. To this end, we propose a novel approach that can exploit more effectively the rich video information, by accounting for the role that the changing pose/viewpoint factor plays in the sequences matching process. Specifically, our approach consists of two components. The first one attempts to complement the original pose-incomplete information carried by the sequences with synthetic GAN-generated images, and fuse their feature vectors into a more discriminative viewpoint-insensitive embedding, namely Weighted Fusion (WF). Another one performs an explicit pose-based alignment of sequence pairs to promote coherent feature matching, namely Weighted-Pose Regulation (WPR). Extensive experiments on two large video-based benchmark datasets show that our approach outperforms considerably existing methods.

Introduction

Person re-identification (Re-ID) is the critical task of identifying the same individual across non-overlapping surveillance camera networks. While video sequences provide significantly more data than single images, they introduce unique challenges, primarily pose incompleteness and viewpoint misalignment.

In practical scenarios, a camera may only capture a person from a single angle (e.g., a back view), while a different camera captures them from another (e.g., a side view). Standard methods often average features across all frames, which introduces noise by comparing disparate viewpoints. To achieve robust matching, it is necessary to compare "apples to apples"—matching specific poses across sequences—and to account for viewpoints that were never captured by the cameras.

Methodology

The proposed framework, Pose-Driven Sequence Regulation (PDSR), addresses these challenges through two primary mechanisms: Weighted Fusion (WF) and Weighted-Pose Regulation (WPR).

Weighted Fusion (WF)

To mitigate pose incompleteness, a Generative Adversarial Network (GAN) is employed to synthesize images of the target individual in eight canonical poses (e.g., front, 45-degree, side, back). During inference, these synthetic images are processed through a Residual CNN to extract features. These features are then fused with those from the original video sequence to create a "viewpoint-insensitive" embedding. This process ensures the final representation includes information from all angles, even those missing from the raw footage.

System architecture overview showing GAN-based image generation and feature fusion.

Weighted-Pose Regulation (WPR)

WPR facilitates explicit spatial alignment. Rather than treating video sequences as monolithic entities, WPR decomposes them into pose-specific sub-sequences. If a sequence lacks a particular pose, the GAN-generated synthetic version is used to fill the gap. This allows the system to perform coherent, pose-to-pose matching between query and gallery sequences, avoiding the "feature washing" effect common in simple temporal averaging.

Pose-Driven Sequence Regulation Workflow

The PDSR architecture utilizes a lightweight residual CNN for feature extraction and a deformable GAN capable of global morphological transformations. This allows the model to change a person's pose while strictly preserving their identity.

Flowchart of the PDSR process, detailing pose alignment and missing pose synthesis.

As shown in the workflow, the system aligns the gallery and query sequences by pose. The GAN is leveraged at test time to generate missing viewpoints, which are then integrated into the final pose-regularized feature map sequences. Interestingly, research indicated that using synthetic images during training introduced too much noise; however, using them to regulate the matching process at inference time proved highly effective.

Experimental Results

The model was evaluated on two major video Re-ID benchmarks: MARS (Motion Analysis and Re-identification Set) and DukeMTMC-VideoReID. The GAN component was pre-trained on the Market-1501 dataset to ensure high-quality pose generation.

  • MARS: The approach achieved a Rank-1 accuracy of 75.76% and a mean Average Precision (mAP) of 60.57%. This represents a significant improvement over previous state-of-the-art methods like P-QAN (+2% Rank-1 and +8.9% mAP).
  • DukeMTMC-VideoReID: The model reached 82.22% Rank-1 accuracy, outperforming semi-unsupervised methods like ETAP-Net by nearly 10%.

Ablation studies confirmed that both Weighted Fusion and Weighted-Pose Regulation contribute independently to these gains, validating the necessity of both data completion and spatial alignment.

Conclusion

PDSR demonstrates that treating video Re-ID as a data-completion and alignment problem significantly improves accuracy. By using GANs to "hallucinate" missing viewpoints, the system provides a holistic 360-degree signature of an individual. While the approach relies on the accuracy of external Human Pose Estimators (HPE) and must manage the computational overhead of GAN generation at test time, it sets a new standard for handling unregulated video sequences in surveillance applications.

Summary updated

Abstract

Video-based person re-identification deals with the inherent difficulty of matching unregulated sequences with different length and with incomplete target pose/viewpoint structure. Common approaches operate either by reducing the problem to the still images case, facing a significant information loss, or by exploiting inter-sequence temporal dependencies as in Siamese Recurrent Neural Networks or in gait analysis. However, in all cases, the inter-sequences pose/viewpoint misalignment is not considered, and the existing spatial approaches are mostly limited to the still images context. To this end, we propose a novel approach that can exploit more effectively the rich video information, by accounting for the role that the changing pose/viewpoint factor plays in the sequences matching process. Specifically, our approach consists of two components. The first one attempts to complement the original pose-incomplete information carried by the sequences with synthetic GAN-generated images, and fuse their feature vectors into a more discriminative viewpoint-insensitive embedding, namely Weighted Fusion (WF). Another one performs an explicit pose-based alignment of sequence pairs to promote coherent feature matching, namely Weighted-Pose Regulation (WPR). Extensive experiments on two large video-based benchmark datasets show that our approach outperforms considerably existing methods.
BibTeX
@inproceedings{borgia2019gan,
  author = {Alessandro Borgia and Yang Hua and Elyor Kodirov and Neil Martin Robertson},
  title = {{GAN-based Pose-aware Regulation for Video-based Person Re-identification}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2019},
  url = {https://openreview.net/forum?id=Q_xiIa3ND5h},
}
Figure