TL;DR
This paper introduces CIST-GCN, a novel graph convolutional network for human motion forecasting that not only predicts future poses but also provides interpretable insights into its predictions. By incorporating learnable adjacency matrices and feature importance vectors, CIST-GCN offers a more transparent understanding of motion dynamics, outperforming existing methods in accuracy and robustness across multiple datasets.
What Problem Does This Paper Address?
Human motion forecasting is crucial for applications like autonomous driving and robotics, but it remains a challenging problem. Existing methods, while improving prediction accuracy, often operate as "black boxes," making it difficult to understand why a particular motion is predicted. This lack of interpretability hinders trust and debugging, especially when dealing with complex or unexpected movements. Furthermore, understanding the relationships between body joints and temporal frames is key to both accurate prediction and preliminary interpretation of body behavior.
Methods
The core of this work is the Context-based Interpretable Spatio-Temporal Graph Convolutional Network (CIST-GCN). It builds upon Graph Convolutional Networks (GCNs) by incorporating several key innovations:
- Spatio-Temporal GCN (DST-GCN): This module processes the inherent spatial (joint-to-joint) and temporal (frame-to-frame) relationships in motion sequences. It uses separable spatial and temporal graph convolutions.
- Dynamic Adjacency Encoder (DAE): Instead of fixed adjacency matrices, CIST-GCN uses a DAE to learn sample-specific adjacency matrices. This allows the model to adapt its understanding of joint relationships based on the input motion.
- Atrous Pyramid Temporal Convolutional Network (APTCN): The decoder part of the architecture uses an APTCN, inspired by DeepLabv3+, to capture multi-scale temporal information with different dilation rates.
- Context Network (ConNet): This network extracts feature importance vectors, providing insights into which parts of the motion are most influential for the prediction.
- Gating Network (GaNet): A lightweight gating network weighs the outputs from the DST-GCN blocks, further refining the prediction and contributing to interpretability.
The model also incorporates input features that include joint positions, velocities, and accelerations, providing a richer representation of motion dynamics.
Datasets
The proposed CIST-GCN model was evaluated on several widely-used datasets:
- Human3.6M: A large dataset for 3D human pose estimation and motion prediction, containing 15 different actions.
- AMASS: A comprehensive collection of 18 motion capture datasets, providing diverse motion sequences.
- 3DPW: A dataset featuring both indoor and outdoor actions, used to test generalization.
- ExPI: A dataset specifically designed for multi-person motion interaction, featuring dance sequences.
The authors also mention experiments with "out-of-distribution" (OOD) data, including rotations and noise, to test the model's robustness.
Key Insights and Ideas
- Interpretability is Achievable in Motion Forecasting: The paper demonstrates that it's possible to build motion forecasting models that are not only accurate but also provide meaningful interpretations of their predictions, moving beyond the "black box" paradigm.
- Sample-Specific Adjacency Matrices are Crucial: Learning dynamic adjacency matrices that adapt to each input sequence significantly improves the model's ability to capture nuanced joint relationships, leading to better predictions.
- Feature Importance Vectors Offer Actionable Insights: The generated feature importance vectors and adjacency matrices can reveal how the model perceives motion, helping to identify key joints and temporal dependencies that drive predictions. This can be useful for understanding complex or unexpected movements.
- Robustness Through Data Augmentation: The inclusion of data augmentation techniques like random rotations and noise significantly enhances the model's resilience to adversarial attacks and variations in input data.
Key Results
- State-of-the-Art Performance: CIST-GCN consistently achieves comparable or superior performance to existing state-of-the-art (SOTA) models across Human3.6M, AMASS, 3DPW, and ExPI datasets, particularly in terms of Mean Per Joint Position Error (MPJPE).
- Robustness: The model demonstrates strong robustness against random rotation attacks and noise, outperforming other methods in these scenarios.
- Lightweight Architecture: Despite its advanced capabilities, CIST-GCN is relatively lightweight in terms of parameters and computational complexity compared to many SOTA models, especially when considering its interpretability features. For instance, the M32 model has a complexity of 0.76M, significantly less than some competitors.
- Qualitative Interpretations: The paper provides visual examples (e.g., Figure 2, 3, 5, 6) showcasing how the learned adjacency matrices and feature importance maps can be used to understand motion patterns, identify difficult-to-predict samples, and even reveal similarities between different motion classes.
Limitations and Future Work
- Computational Cost of Interpretability: While the model is efficient overall, the interpretability features (matrix multiplications and linear layers for feature importance) do increase the FLOPs compared to simpler models.
- M64 Model Instability: The M64 configuration, while achieving the lowest error, experienced training instability due to potential overflow or large gradients from the sum operator in DST-GCN blocks.
- Future Work: The authors plan to extend their study to adversarial attacks and OOD scenarios for a deeper understanding of feature maps. They also aim to optimize model complexity further. Additionally, they suggest further research is needed to fully comprehend which layers within the network are responsible for transforming input displacements into output displacements.
Summary updated