Multi-Level Graph Encoding with Structural-Collaborative Relation Learning for Skeleton-Based Person Re-Identification

Haocong Rao, Shihao Xu, Xiping Hu, Jun Cheng, Bin HuIJCAI 2021 · DOI

TL;DR

This paper introduces MG-SCR, a novel multi-level graph encoding approach for skeleton-based person re-identification (Re-ID). By learning structural and collaborative relationships between body components across different graph levels, MG-SCR significantly improves Re-ID accuracy, even outperforming multi-modal methods that use RGB or depth data.

What Problem Does This Paper Address?

Person re-identification (Re-ID) is crucial for tracking individuals across different cameras and scenes. While RGB and depth-based methods exist, skeleton-based Re-ID offers advantages like smaller data size and robustness to scale and view variations. However, existing skeleton-based methods often rely on hand-crafted features or simple dynamic modeling, failing to fully exploit the rich structural and relational information inherent in human skeletons. This paper aims to bridge this gap by proposing a method that explicitly models these complex relationships across multiple levels of skeleton representation.

Methods

The core of the proposed method, MG-SCR, lies in its multi-level graph construction and a novel learning paradigm:

  • Multi-Level Graph Construction: Skeletons are represented as three levels of graphs: joint-level (individual joints), part-level (groups of joints like limbs), and body-level (higher-level components). This allows for capturing relationships at different granularities.
  • Multi-Head Structural Relation Layer (MSRL): This layer learns "structural relations" between neighboring nodes within each graph level. It uses a graph attention mechanism to assign weights to correlated features, effectively aggregating key information from connected body parts. Multiple "heads" are used to capture diverse structural relationships independently.
  • Cross-Level Collaborative Relation Layer (CCRL): This novel layer infers "collaborative relations" between body components across different graph levels (e.g., how a leg joint relates to a hip component). This captures the dynamic cooperation between body parts during movement, which is crucial for gait patterns.
  • Multi-Level Graph Feature Fusion: Features from the different graph levels are fused using a weighted sum to create a comprehensive multi-level graph representation.
  • Graph Dynamics Encoding (SSP Pre-training): A self-supervised sparse sequential prediction task is introduced. This involves predicting future skeleton poses from sparsely sampled subsequences. This pre-training helps the model learn temporal dynamics and high-level semantics of skeleton sequences, improving robustness.
  • Recognition Layer: Encoded graph states from the LSTM are averaged and fed into a recognition layer for final person Re-ID prediction.

Datasets

The paper evaluates its approach on four public skeleton-based Re-ID datasets:

  • IAS-Lab
  • BIWI
  • KS20
  • KGBD

Additionally, to assess performance with estimated skeletons, they use the large-scale CASIA B dataset, which provides RGB videos. For CASIA B, 3D skeletons are extracted using pre-trained pose estimation models.

Key Insights and Ideas

  • Hierarchical Skeleton Representation is Key: Representing skeletons at multiple levels (joint, part, body) unlocks richer information than a single-level representation.
  • Structural and Collaborative Relations are Complementary: Explicitly modeling both how parts connect structurally and how they cooperate dynamically leads to more discriminative features. CCRL is a novel contribution here.
  • Self-Supervised Pre-training Boosts Performance: The SSP task effectively leverages unlabeled skeleton data to learn temporal dynamics, significantly improving the model's ability to capture gait and motion patterns.
  • Averaging Predictions Improves Robustness: Using the average prediction across all encoded states (rather than just the last one) helps mitigate the impact of noisy frames and leads to more stable sequence-level Re-ID.

Key Results

MG-SCR achieves state-of-the-art performance on skeleton-based Re-ID across multiple datasets.

  • It significantly outperforms existing skeleton-based methods, including hand-crafted feature methods (up to 49.4% Rank-1 improvement) and recent deep learning models (up to 56.5% Rank-1 improvement).
  • Crucially, MG-SCR also surpasses many multi-modal methods that utilize RGB or depth information, demonstrating the power of its skeleton-centric approach. For instance, on the BIWI dataset, it achieves a Rank-1 accuracy of 61.6%, outperforming many multi-modal baselines.
  • The model shows strong performance even with 3D skeletons estimated from RGB videos (CASIA B), outperforming methods like PoseGait by a large margin (up to 61.5% Rank-1 accuracy on some views).

Limitations and Future Work

  • Limitations: The paper acknowledges that while MG-SCR performs well, it relies on accurate 3D skeleton data. The performance might degrade if the skeleton estimation is noisy or inaccurate. The paper also notes that some multi-modal methods achieve higher accuracy on specific datasets (e.g., IAS-B with PCM + Skeleton), suggesting potential for further integration of modalities.
  • Future Work: The authors suggest that their approach is promising for large-scale RGB-based datasets and can be applied to general settings with varying views. Further exploration into combining MG-SCR with other modalities or improving skeleton estimation robustness could be future directions. The visualization of collaborative relations also opens avenues for deeper analysis of human motion dynamics.

Summary updated

BibTeX
@inproceedings{rao2021multi,
  author = {Haocong Rao and Shihao Xu and Xiping Hu and Jun Cheng and Bin Hu},
  title = {{Multi-Level Graph Encoding with Structural-Collaborative Relation Learning for Skeleton-Based Person Re-Identification}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.24963/ijcai.2021/135},
}
Figure