Abstract
Video-based person re-identification deals with the inherent difficulty of matching unregulated sequences with different length and with incomplete target pose/viewpoint structure. Common approaches operate either by reducing the problem to the still images case, facing a significant information loss, or by exploiting inter-sequence temporal dependencies as in Siamese Recurrent Neural Networks or in gait analysis. However, in all cases, the inter-sequences pose/viewpoint misalignment is not considered, and the existing spatial approaches are mostly limited to the still images context. To this end, we propose a novel approach that can exploit more effectively the rich video information, by accounting for the role that the changing pose/viewpoint factor plays in the sequences matching process. Specifically, our approach consists of two components. The first one attempts to complement the original pose-incomplete information carried by the sequences with synthetic GAN-generated images, and fuse their feature vectors into a more discriminative viewpoint-insensitive embedding, namely Weighted Fusion (WF). Another one performs an explicit pose-based alignment of sequence pairs to promote coherent feature matching, namely Weighted-Pose Regulation (WPR). Extensive experiments on two large video-based benchmark datasets show that our approach outperforms considerably existing methods.
Introduction
Person re-identification (Re-ID) is the critical task of identifying the same individual across non-overlapping surveillance camera networks. While video sequences provide significantly more data than single images, they introduce unique challenges, primarily pose incompleteness and viewpoint misalignment.
In practical scenarios, a camera may only capture a person from a single angle (e.g., a back view), while a different camera captures them from another (e.g., a side view). Standard methods often average features across all frames, which introduces noise by comparing disparate viewpoints. To achieve robust matching, it is necessary to compare "apples to apples"—matching specific poses across sequences—and to account for viewpoints that were never captured by the cameras.
Methodology
The proposed framework, Pose-Driven Sequence Regulation (PDSR), addresses these challenges through two primary mechanisms: Weighted Fusion (WF) and Weighted-Pose Regulation (WPR).
Weighted Fusion (WF)
To mitigate pose incompleteness, a Generative Adversarial Network (GAN) is employed to synthesize images of the target individual in eight canonical poses (e.g., front, 45-degree, side, back). During inference, these synthetic images are processed through a Residual CNN to extract features. These features are then fused with those from the original video sequence to create a "viewpoint-insensitive" embedding. This process ensures the final representation includes information from all angles, even those missing from the raw footage.
Weighted-Pose Regulation (WPR)
WPR facilitates explicit spatial alignment. Rather than treating video sequences as monolithic entities, WPR decomposes them into pose-specific sub-sequences. If a sequence lacks a particular pose, the GAN-generated synthetic version is used to fill the gap. This allows the system to perform coherent, pose-to-pose matching between query and gallery sequences, avoiding the "feature washing" effect common in simple temporal averaging.
Pose-Driven Sequence Regulation Workflow
The PDSR architecture utilizes a lightweight residual CNN for feature extraction and a deformable GAN capable of global morphological transformations. This allows the model to change a person's pose while strictly preserving their identity.
As shown in the workflow, the system aligns the gallery and query sequences by pose. The GAN is leveraged at test time to generate missing viewpoints, which are then integrated into the final pose-regularized feature map sequences. Interestingly, research indicated that using synthetic images during training introduced too much noise; however, using them to regulate the matching process at inference time proved highly effective.
Experimental Results
The model was evaluated on two major video Re-ID benchmarks: MARS (Motion Analysis and Re-identification Set) and DukeMTMC-VideoReID. The GAN component was pre-trained on the Market-1501 dataset to ensure high-quality pose generation.
- MARS: The approach achieved a Rank-1 accuracy of 75.76% and a mean Average Precision (mAP) of 60.57%. This represents a significant improvement over previous state-of-the-art methods like P-QAN (+2% Rank-1 and +8.9% mAP).
- DukeMTMC-VideoReID: The model reached 82.22% Rank-1 accuracy, outperforming semi-unsupervised methods like ETAP-Net by nearly 10%.
Ablation studies confirmed that both Weighted Fusion and Weighted-Pose Regulation contribute independently to these gains, validating the necessity of both data completion and spatial alignment.
Conclusion
PDSR demonstrates that treating video Re-ID as a data-completion and alignment problem significantly improves accuracy. By using GANs to "hallucinate" missing viewpoints, the system provides a holistic 360-degree signature of an individual. While the approach relies on the accuracy of external Human Pose Estimators (HPE) and must manage the computational overhead of GAN generation at test time, it sets a new standard for handling unregulated video sequences in surveillance applications.
Summary updated