Adaptive Graph Representation Learning for Video Person Re-identification

Yiming Wu, Omar El Farouk Bourahla, Xi Li, Fei Wu, Qi TianIEEE TIP 2019 · DOI · OpenReview

TL;DR

This paper introduces an "Adaptive Graph Representation Learning" (AGRL) method for video person re-identification (Re-ID). AGRL constructs a dynamic graph that captures relationships between different body parts across video frames, leading to more robust feature representations and improved Re-ID performance, especially in challenging scenarios like occlusion.

What Problem Does This Paper Address?

Video person re-identification (Re-ID) aims to identify the same person across different video sequences. While deep learning has made significant strides, accurately re-identifying individuals in videos remains challenging due to variations in pose, illumination, occlusion, and viewpoint. Existing methods often struggle to effectively model the intricate relationships between different body parts within a person, which are crucial for robust recognition, especially when parts of the body are occluded or distorted. This paper addresses the limitation of previous part-based approaches that often ignore these inter-part correlations, proposing a method to explicitly model and leverage these relationships for better video Re-ID.

Methods

The core of the proposed method is an Adaptive Graph Representation Learning (AGRL) scheme. This involves several key components:

  • Adaptive Structure-Aware Adjacency Graph: Instead of a fixed graph structure, AGRL dynamically constructs a graph where nodes represent spatial regions within video frames. This graph has two types of edges:
  • Pose Alignment Connection: Connects regions that contain the same human body part (e.g., head, trunk, legs) based on pose estimation. This helps align spatial regions across frames.
  • Feature Affinity Connection: Connects regions based on the visual similarity of their extracted features. This captures semantic relationships between regions. These two connections are combined to form an adaptive adjacency graph.

  • Graph Feature Propagation: A Graph Neural Network (GNN) is used to iteratively refine the regional features by aggregating contextual information from neighboring nodes on the constructed graph. This process enhances discriminative features and weakens noisy ones.

  • Temporal Attention Module: After graph-based feature propagation, a spatio-temporal attention mechanism is applied to generate a discriminative video representation.
  • Temporal Resolution-Aware Regularization: To ensure compact and discriminative representations, a novel regularization is introduced. This enforces consistency among video representations derived from subsequences sampled at different temporal resolutions, making the model robust to temporal variations.
  • Loss Functions: The model is trained using a combination of cross-entropy loss and a soft-hard triplet loss, applied to both the global features (from a standard CNN branch) and the graph-enhanced features.

Datasets

The proposed method was evaluated on four widely used video person Re-ID benchmarks:

  • iLIDS-VID
  • PRID2011
  • MARS
  • DukeMTMC-VideoReID

The paper does not introduce any new datasets.

Key Insights and Ideas

  • Modeling Inter-Part Relationships is Crucial: The paper strongly emphasizes that simply extracting local features is insufficient. Explicitly modeling the relationships between different body parts (e.g., how the head relates to the trunk) is vital for robust Re-ID, especially when dealing with occlusions.
  • Dynamic Graphs Outperform Static Ones: By creating an "adaptive" graph that considers both pose and feature similarity, the method can better capture the nuanced relationships between regions, leading to more effective feature propagation.
  • Pose Information as a Structural Prior: Leveraging human pose estimation to guide graph construction provides a strong structural prior, helping to align relevant parts across frames and improving robustness to pose variations.
  • Temporal Consistency is Key for Video: The temporal resolution-aware regularization highlights the importance of ensuring that representations are consistent regardless of how the video is sampled or segmented, which is a key challenge in video analysis.

Key Results

The proposed AGRL method achieved state-of-the-art performance on the MARS and DukeMTMC-VideoReID datasets.

  • On MARS, the method achieved Rank-1 accuracy of 89.8% and mAP of 81.1%, outperforming the previous best approach by 0.6% and 0.7% respectively.
  • On DukeMTMC-VideoReID, it achieved Rank-1 accuracy of 97.0% and mAP of 95.4%, surpassing the state-of-the-art STA by a significant margin.
  • On the smaller datasets iLIDS-VID and PRID2011, the method also demonstrated competitive performance, achieving Rank-1 accuracies of 84.5% and 94.6% respectively, outperforming methods that do not use optical flow and remaining competitive with those that do.

Qualitative results using Grad-CAM visualizations showed that the graph-based approach effectively reduces the focus on distracting background elements and is less influenced by occlusions compared to a baseline model.

Limitations and Future Work

  • Limitations:
    • The reliance on accurate human pose estimation is a potential limitation. Errors in pose estimation could negatively impact the graph construction.
  • While the method shows robustness to occlusion, extreme or complete occlusion of key body parts might still pose challenges.
  • Future Work:
    • The paper suggests that further exploration of graph structures and propagation mechanisms could lead to even better performance.
  • Investigating more sophisticated ways to combine pose and feature affinity information could be a fruitful direction.
  • Extending the approach to other video-based tasks beyond Re-ID could be explored.

Summary updated

Abstract

Recent years have witnessed the remarkable progress of applying deep learning models in video person re-identification (Re-ID). A key factor for video person Re-ID is to effectively construct discriminative and robust video feature representations for many complicated situations. Part-based approaches employ spatial and temporal attention to extract representative local features. While correlations between parts are ignored in the previous methods, to leverage the relations of different parts, we propose an innovative adaptive graph representation learning scheme for video person Re-ID, which enables the contextual interactions between relevant regional features. Specifically, we exploit the pose alignment connection and the feature affinity connection to construct an adaptive structure-aware adjacency graph, which models the intrinsic relations between graph nodes. We perform feature propagation on the adjacency graph to refine regional features iteratively, and the neighbor nodes' information is taken into account for part feature representation. To learn compact and discriminative representations, we further propose a novel temporal resolution-aware regularization, which enforces the consistency among different temporal resolutions for the same identities. We conduct extensive evaluations on four benchmarks, i.e. iLIDS-VID, PRID2011, MARS, and DukeMTMC-VideoReID, experimental results achieve the competitive performance which demonstrates the effectiveness of our proposed method. The code is available at https://github.com/weleen/AGRL.pytorch.
BibTeX
@article{wu2019adaptive,
  author = {Yiming Wu and Omar El Farouk Bourahla and Xi Li and Fei Wu and Qi Tian},
  title = {{Adaptive Graph Representation Learning for Video Person Re-identification}},
  journal = {IEEE Trans. Image Process.},
  year = {2019},
  doi = {10.1109/tip.2020.3001693},
}
Figure