Abstract
Synthesizing long-duration human action sequences is challenging due to error accumulation in traditional autoregressive models. This paper introduces the Convolutional Sequence Generation Network (CSGN), which generates entire sequences globally using Gaussian Process priors and hierarchical graph convolutions. CSGN ensures temporal smoothness and global structural coherence, significantly outperforming existing methods on complex datasets like the newly introduced MikuDance.
Introduction
Synthesizing realistic human motion, particularly for long-duration sequences like a multi-minute dance, presents a significant challenge in computer vision. Traditional approaches typically rely on autoregressive models such as LSTMs or GRUs, which predict the next frame based on a limited history. However, these methods suffer from two critical drawbacks: error accumulation, where small inaccuracies snowball into "drifting" or frozen poses, and a lack of global structure, as they struggle to capture multi-scale temporal dependencies.
The CSGN Framework
To address these limitations, the authors propose the Convolutional Sequence Generation Network (CSGN). Unlike step-by-step predictors, CSGN treats action synthesis as a global generation task, akin to hierarchical image generation.
The core of CSGN lies in its hierarchical transformation process. It begins by sampling a latent variable sequence from a Gaussian Process (GP) prior. By employing a Radial Basis Function (RBF) kernel, the model enforces temporal correlations across various scales, ensuring the generated motion is inherently smooth. This latent sequence is then progressively upsampled in both time (increasing frame count) and space (increasing skeletal joint density).
Hierarchical Graph Pyramids
Because human skeletons are structured as graphs rather than regular grids, CSGN utilizes "graph pyramids" to perform principled upsampling. The network moves from a coarse representation (e.g., a single point) to a high-fidelity 46-joint body. Furthermore, the model employs spatially-varying graph convolutions, which use distinct parameters for different body parts to respect their unique movement patterns.
Applications and Versatility
A key feature of CSGN is its bidirectional mapping. By training an inverse mapping network ($H$), the system can project existing skeleton sequences back into the latent space. This enables a variety of downstream tasks beyond simple generation.
As shown above, the model can perform probabilistic prediction and action complementation. In complementation, the model can realistically fill in missing transitions between two disjoint segments of a dance, maintaining global coherence that autoregressive models often lose.
Datasets and Evaluation
The model was evaluated on the standard NTU-RGB+D dataset and a new, high-quality dataset called MikuDance. MikuDance consists of 201 long-duration dance sequences (3–5 minutes each), totaling 10 hours of motion with 46 joints and quaternion-based rotations.
Quantitatively, CSGN achieved a Mean Ensemble FID of 23.49 on MikuDance, significantly outperforming state-of-the-art autoregressive models like ERD (47.49) and acLSTM (50.96). Qualitatively, CSGN demonstrated remarkable stability, maintaining diverse and meaningful motion for sequences up to 60 seconds (1800 frames), whereas traditional models often "froze" or became repetitive after just a few seconds.
Conclusion and Future Work
CSGN demonstrates that global convolutional generation is a powerful alternative to recurrence for long-duration action synthesis. While the model occasionally produces unnatural poses or "foot sliding" due to its purely data-driven nature, these issues could be mitigated in the future by integrating explicit physical constraints or bio-mechanical priors into the convolutional layers.
Reference: Yan, S., Li, Z., Xiong, Y., Yan, H., & Lin, D. (2019). Convolutional Sequence Generation for Skeleton-Based Action Synthesis. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV).
Summary updated