TL;DR
This paper introduces Kinetic-GAN, a novel Generative Adversarial Network (GAN) that uses Graph Convolutional Networks (GCNs) to synthesize realistic and diverse human action sequences from skeleton data. It addresses the limitations of existing methods by enabling high-quality, action-conditioned generation of up to 120 different actions, improving both sample quality and diversity.
What Problem Does This Paper Address?
Synthesizing realistic human actions from skeleton data is a challenging task. Existing methods often struggle with generating diverse actions, accurately conditioning specific actions, and capturing the long-term temporal dependencies of movements. Many approaches rely on manually structured vector sequences, which can be suboptimal for capturing the inherent graph structure of the human skeleton. Furthermore, the limited availability of diverse, high-quality action data hinders the development of robust human behavior analysis systems. This paper aims to overcome these limitations by proposing a more scalable and effective approach to human action synthesis.
Methods
The core of Kinetic-GAN is a GAN architecture that leverages Graph Convolutional Networks (GCNs). Instead of treating skeleton data as simple sequences of coordinates, GCNs are used to exploit the inherent graph structure of the human body (joints as nodes, bones as edges).
- Spatiotemporal Graph Convolution: The generator and discriminator employ spatiotemporal graph convolutions. This involves:
- Spatial Graph Convolution: Captures relationships between joints within a single frame, using adjacency matrices to define connections. Learnable weight matrices are introduced to adapt these spatial weights across different layers.
- Temporal Graph Convolution: A 1D convolution applied across consecutive frames to capture temporal dependencies between joints.
- Generator Architecture: The generator takes a latent code
zand an action class embeddingyas input. It uses a non-linear mapping network to transformzinto an intermediate latent spaceW, aiming to disentangle factors of variation. The generator then gradually upsamples the skeleton graph spatially and temporally. - Discriminator Architecture: The discriminator performs the inverse operation, downsampling the skeleton graph spatially and temporally to distinguish between real and synthesized sequences.
- Conditional Adversarial Training: The model is trained using a WGAN-GP objective, incorporating the action class embedding
yto condition the generated actions. - Stochastic Variation: To enhance diversity, random noise is injected into each joint after each spatiotemporal graph convolution in the generator. This allows for adaptive learning of stochastic variations without compromising the skeleton structure.
- Truncation Trick: To improve sample quality, a truncation trick is applied to the intermediate latent space
Wduring inference, scaling the deviation of latent points from the center mass.
Datasets
The paper evaluates Kinetic-GAN on three well-known datasets:
- NTU RGB+D [31]: A large-scale dataset with 60 action classes and 3D skeleton data from 40 volunteers.
- NTU-120 RGB+D [25]: An extended version of NTU RGB+D with 120 action classes, captured in more diverse setups.
- Human3.6M [18]: A dataset with 2D human motions and 15 body joints.
The paper also mentions NTU-2D RGB+D, a curated version for local movement settings.
Key Insights and Ideas
- Graph Structure is Key: Explicitly modeling the human skeleton as a graph using GCNs is crucial for capturing the inherent structural and temporal relationships, outperforming methods that rely on vector sequences.
- Disentangled Latent Space for Diversity: The introduction of a mapping network to create an intermediate latent space
Whelps disentangle factors of variation, leading to more diverse and controllable action generation. - Stochastic Noise for Adaptive Variation: Injecting noise at each layer allows for adaptive learning of variations, improving sample diversity without sacrificing the integrity of the action sequence.
- Global Movement Synthesis is Achievable: The proposed architecture successfully synthesizes actions in global movement settings, a significant improvement over previous methods often limited to local movements.
- Scalability to Many Actions: Kinetic-GAN demonstrates the ability to synthesize a large number of distinct actions (up to 120), a substantial leap from prior work that could only handle around 10.
Key Results
- State-of-the-Art Performance: Kinetic-GAN achieves state-of-the-art results on NTU RGB+D and NTU-120 RGB+D datasets in terms of FID and MMD scores, indicating higher quality and better distribution matching of synthesized actions.
- 120 Action Classes: The model can generate up to 120 different actions, a significant increase compared to previous methods.
- Improved Temporal Length: The model can generate long-term action sequences up to 1024 frames (34 seconds) on Human3.6M, maintaining quality and avoiding pose freezing issues common in autoregressive models.
- Reduced Artefacts: By carefully managing batch normalization, the model reduces spatial artefacts often seen in GAN-generated images.
Limitations and Future Work
- Artefact Reduction Trade-off: While reducing batch normalization helps eliminate artefacts, it can impact training stability and the ability to generate desirable actions. The paper proposes a regularized approach to balance these.
- Truncation Trick Trade-off: The truncation trick improves sample quality but can lead to a decrease in variation, requiring careful selection of the truncation threshold.
- Future Work: The authors suggest exploring interpretable latent space directions and style transfer between action samples, leveraging the generation of human actions directly from the latent space. Further research into bidirectional temporal dependency and extending the model to other domains is also implied.
Summary updated