GAN-Based Pose-Aware Regulation for Video-Based Person Re-Identification

Alessandro Borgia, Yang Hua, Elyor Kodirov, Neil Martin Robertson

TL;DR

This paper introduces Pose-Driven Sequence Regulation (PDSR), a framework that improves video-based person re-identification by using GANs to "fill in the blanks" of missing viewpoints. By synthesizing a complete set of canonical poses for every person and explicitly aligning these poses during matching, the authors achieve state-of-the-art results on major benchmarks like MARS and DukeMTMC.

What Problem Does This Paper Address?

Person re-identification (Re-ID) is the task of spotting the same individual across different, non-overlapping surveillance cameras. While video-based Re-ID provides more data than single-image methods, it faces a major hurdle: pose incompleteness and misalignment.

In the real world, a camera might only capture a person walking away (back view), while another camera captures them from the side. Most existing methods simply average the features of all frames in a video sequence. The authors argue this is sub-optimal because it ignores the spatial context—comparing a "back view" feature to a "side view" feature is inherently noisy. To truly match two videos, you need a way to compare "apples to apples" (front-to-front, side-to-side) and account for the viewpoints that the camera missed.

Methods

The authors propose a two-pronged approach called Pose-Driven Sequence Regulation (PDSR):

  1. Weighted Fusion (WF): The researchers use a Generative Adversarial Network (GAN) to generate synthetic images of the target person in eight "canonical" poses (e.g., front, 45-degree, side, back). At inference time, these synthetic images are passed through a CNN to extract features, which are then fused with the features from the original, incomplete video. This creates a "viewpoint-insensitive" embedding that represents the person from all angles, even those the camera didn't see.

  2. Weighted-Pose Regulation (WPR): This component handles the alignment. Instead of comparing entire video sequences as monolithic blobs, WPR breaks them down by pose. It groups frames into pose-specific sub-sequences. If a specific pose is missing in the real video, the GAN-generated version is used to plug the gap. When matching two people, the system compares them pose-by-pose, ensuring a coherent spatial comparison.

The Architecture: They utilize a light-weight residual CNN for feature extraction and a deformable GAN architecture capable of handling global morphological transformations (changing a person's pose while keeping their identity).

Datasets

The model was evaluated on two massive video Re-ID benchmarks:

  • MARS (Motion Analysis and Re-identification Set): One of the largest datasets with 1,261 identities and over 20,000 video sequences.
  • DukeMTMC-VideoReID: A large-scale dataset derived from the DukeMTMC tracking dataset, featuring 1,404 identities captured by 8 disjoint cameras.

Note: The GAN was pre-trained on the Market-1501 image dataset, as it provided cleaner training samples for pose generation than the noisy video tracklets.

Key Insights and Ideas

  • Synthesizing Completeness: The "aha moment" here is treating video Re-ID as a data-completion problem. By using a GAN to hallucinate missing viewpoints, they provide the classifier with a holistic 360-degree signature of the person.
  • Inference-Time Augmentation: Unlike many GAN-based approaches that use synthetic data only for training, this method uses the GAN at test time. Interestingly, the authors found that adding synthetic images to the training set actually hurt performance due to the noise introduced by the GAN. However, using them to regulate the matching process at the end proved highly effective.
  • Pose-to-Pose Matching: By explicitly aligning sequences by pose, the model avoids the "feature washing" effect that happens when you average a sequence containing wildly different viewpoints.

Key Results

The PDSR approach significantly outperformed existing state-of-the-art (SOTA) methods:

  • MARS: Achieved a Rank-1 accuracy of 75.76% and a mean Average Precision (mAP) of 60.57%. This represents a +2% Rank-1 and a massive +8.9% mAP improvement over the previous best method (P-QAN).
  • DukeMTMC-VideoReID: Reached 82.22% Rank-1 accuracy, outperforming the semi-unsupervised ETAP Net by nearly 10%.
  • Ablation Studies: The results showed that both Weighted Fusion and Weighted-Pose Regulation contribute independently to the performance boost, confirming that both "completing" the data and "aligning" it are necessary.

Limitations and Future Work

  • GAN Artifacts: The authors acknowledge that GAN-generated images are still noisier than real ones. This is why they use a weighting parameter ($w$) to control how much the synthetic features influence the final embedding.
  • Computational Overhead: Generating eight synthetic images for every query and gallery sequence at test time adds computational cost, though the authors mitigated this by using a relatively shallow, fast CNN for feature extraction.
  • Dependency on Pose Estimation: The system relies on an external Human Pose Estimator (HPE). If the pose estimation fails (e.g., in very low resolution or heavy occlusion), the subsequent GAN generation and alignment will likely suffer. Future work could involve end-to-end training that is more robust to pose-estimation errors.

Summary updated

Abstract

Video-based person re-identification deals with the inherent difficulty of matching sequences with different length, unregulated, and incomplete target pose/viewpoint structure. Common approaches operate either by reducing the problem to the still images case, facing a significant information loss, or by exploiting inter-sequence temporal dependencies as in Siamese Recurrent Neural Networks or in gait analysis. However, in all cases, the inter-sequences pose/viewpoint misalignment is considered, and the existing spatial approaches are mostly limited to the still images context. To this end, we propose a novel approach that can exploit more effectively the rich video information, by accounting for the role that the changing pose/viewpoint factor plays in the sequences matching process. In particular, our approach consists of two components. The first one attempts to complement the original pose-incomplete information carried by the sequences with synthetic GAN-generated images, and fuse their features vectors into a more discriminative viewpoint-insensitive embedding, namely Weighted Fusion (WF). Another one performs an explicit pose-based alignment of sequence pairs to promote coherent feature matching, namely Weighted-Pose Regulation (WPR). Extensive experiments on two large video-based benchmark datasets show that our approach outperforms considerably existing methods.
BibTeX
@inproceedings{borgia2019gan,
  author = {Alessandro Borgia and Yang Hua and Elyor Kodirov and Neil Martin Robertson},
  title = {{GAN-Based Pose-Aware Regulation for Video-Based Person Re-Identification}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2019},
  url = {https://openreview.net/forum?id=Q_xiIa3ND5h},
}

Loading related content...

Figure