TL;DR
This paper introduces MG-SCR, a novel multi-level graph encoding approach for skeleton-based person re-identification (Re-ID). By learning structural and collaborative relationships between body components across different graph levels, MG-SCR significantly improves Re-ID accuracy, even outperforming multi-modal methods that use RGB or depth data.
What Problem Does This Paper Address?
Person re-identification (Re-ID) is crucial for tracking individuals across different cameras and scenes. While RGB and depth-based methods exist, skeleton-based Re-ID offers advantages like smaller data size and robustness to scale and view variations. However, existing skeleton-based methods often rely on hand-crafted features or simple dynamic modeling, failing to fully exploit the rich structural and relational information inherent in human skeletons. This paper aims to bridge this gap by proposing a method that explicitly models these complex relationships across multiple levels of skeleton representation.
Methods
The core of the proposed method, MG-SCR, lies in its multi-level graph construction and a novel learning paradigm:
- Multi-Level Graph Construction: Skeletons are represented as three levels of graphs: joint-level (individual joints), part-level (groups of joints like limbs), and body-level (higher-level components). This allows for capturing relationships at different granularities.
- Multi-Head Structural Relation Layer (MSRL): This layer learns "structural relations" between neighboring nodes within each graph level. It uses a graph attention mechanism to assign weights to correlated features, effectively aggregating key information from connected body parts. Multiple "heads" are used to capture diverse structural relationships independently.
- Cross-Level Collaborative Relation Layer (CCRL): This novel layer infers "collaborative relations" between body components across different graph levels (e.g., how a leg joint relates to a hip component). This captures the dynamic cooperation between body parts during movement, which is crucial for gait patterns.
- Multi-Level Graph Feature Fusion: Features from the different graph levels are fused using a weighted sum to create a comprehensive multi-level graph representation.
- Graph Dynamics Encoding (SSP Pre-training): A self-supervised sparse sequential prediction task is introduced. This involves predicting future skeleton poses from sparsely sampled subsequences. This pre-training helps the model learn temporal dynamics and high-level semantics of skeleton sequences, improving robustness.
- Recognition Layer: Encoded graph states from the LSTM are averaged and fed into a recognition layer for final person Re-ID prediction.
Datasets
The paper evaluates its approach on four public skeleton-based Re-ID datasets:
- IAS-Lab
- BIWI
- KS20
- KGBD
Additionally, to assess performance with estimated skeletons, they use the large-scale CASIA B dataset, which provides RGB videos. For CASIA B, 3D skeletons are extracted using pre-trained pose estimation models.
Key Insights and Ideas
- Hierarchical Skeleton Representation is Key: Representing skeletons at multiple levels (joint, part, body) unlocks richer information than a single-level representation.
- Structural and Collaborative Relations are Complementary: Explicitly modeling both how parts connect structurally and how they cooperate dynamically leads to more discriminative features. CCRL is a novel contribution here.
- Self-Supervised Pre-training Boosts Performance: The SSP task effectively leverages unlabeled skeleton data to learn temporal dynamics, significantly improving the model's ability to capture gait and motion patterns.
- Averaging Predictions Improves Robustness: Using the average prediction across all encoded states (rather than just the last one) helps mitigate the impact of noisy frames and leads to more stable sequence-level Re-ID.
Key Results
MG-SCR achieves state-of-the-art performance on skeleton-based Re-ID across multiple datasets.
- It significantly outperforms existing skeleton-based methods, including hand-crafted feature methods (up to 49.4% Rank-1 improvement) and recent deep learning models (up to 56.5% Rank-1 improvement).
- Crucially, MG-SCR also surpasses many multi-modal methods that utilize RGB or depth information, demonstrating the power of its skeleton-centric approach. For instance, on the BIWI dataset, it achieves a Rank-1 accuracy of 61.6%, outperforming many multi-modal baselines.
- The model shows strong performance even with 3D skeletons estimated from RGB videos (CASIA B), outperforming methods like PoseGait by a large margin (up to 61.5% Rank-1 accuracy on some views).
Limitations and Future Work
- Limitations: The paper acknowledges that while MG-SCR performs well, it relies on accurate 3D skeleton data. The performance might degrade if the skeleton estimation is noisy or inaccurate. The paper also notes that some multi-modal methods achieve higher accuracy on specific datasets (e.g., IAS-B with PCM + Skeleton), suggesting potential for further integration of modalities.
- Future Work: The authors suggest that their approach is promising for large-scale RGB-based datasets and can be applied to general settings with varying views. Further exploration into combining MG-SCR with other modalities or improving skeleton estimation robustness could be future directions. The visualization of collaborative relations also opens avenues for deeper analysis of human motion dynamics.
Summary updated