Learning Bidirectional LSTM Networks for Synthesizing 3D Mesh Animation Sequences

Yi-Ling Qiao, Lin Gao, Yu-Kun Lai, Shihong XiaarXiv 2018 · arXiv

Introduction

Synthesizing high-quality 3D mesh animations is a critical task for the film and gaming industries. However, deep learning approaches face significant hurdles due to the irregular topology of 3D meshes and the high dimensionality of vertex data. Traditional methods often suffer from error accumulation or "freezing," where the model fails to sustain motion over long sequences and settles into a static average shape. This paper presents a novel framework that integrates mesh-based Convolutional Neural Networks (CNNs) with weight-shared Bidirectional LSTMs (BD-LSTMs) to generate stable, realistic 3D animations directly from mesh deformation representations.

Methodology

The proposed architecture addresses spatial geometry and temporal dynamics through a multi-stage process designed to handle the complexities of 3D data.

Spatial Encoding with Mesh CNN

To handle irregular mesh topologies, the framework uses a specialized mesh convolution layer. Instead of raw coordinates, it operates on a shape deformation representation—rotation and scaling matrices derived from deformation gradients. This encoding step reduces high-dimensional mesh data into a compact latent space ($z$), preserving local geometric relationships while making the data more manageable for temporal modeling.

Temporal Modeling with BD-LSTM

The core of the system is a weight-shared Bidirectional LSTM. Unlike standard unidirectional models that only look forward, this architecture processes sequences in both forward and backward directions simultaneously. By sharing weights and employing a bidirectional loss, the model learns a robust motion representation that minimizes drift and stabilizes long-term synthesis.

The architecture of the proposed framework, showing (a) the spatial-temporal cell structure with mesh convolutions and (b) the bidirectional sequence modeling chain.

Experimental Results

The model was evaluated on diverse datasets, including Dyna (human motion) and Horse (galloping sequences), to test its ability to handle both humanoid and non-humanoid topologies.

Qualitative Performance

The framework demonstrates a remarkable ability to maintain motion cycles without "melting" or deforming the mesh unnaturally over time. It can generate long-term sequences (15+ frames) and perform "in-betweening" to complete missing segments between keyframes. Notably, the model learns the periodicity of motion; for example, in a galloping horse, it understands that the legs must return to their starting positions.

Synthesized mesh sequences for human and animal motion, demonstrating the model's ability to capture complex temporal dynamics and maintain geometric integrity.

Quantitative Analysis

In per-vertex position error tests, the BD-LSTM approach significantly outperformed baseline models. On the "Punching" dataset, the proposed method achieved an error of $88 \times 10^{-4}$, compared to $114 \times 10^{-4}$ for unidirectional LSTMs. The inclusion of KL divergence loss and $L_2$ regularization was found to be crucial in preventing the model from settling into a static "average" shape, ensuring the animation remains dynamic.

Key Insights

  • Stability through Bidirectionality: The bidirectional structure forces the network to maintain consistency across the entire sequence, effectively curbing the error accumulation typical of autoregressive models.
  • Bootstrap Power: The model’s accuracy improves as more "bootstrap" (initial) frames are provided, allowing it to better perceive the intended velocity and direction of motion.
  • Diverse Completion: By assigning a random vector to the LSTM state during sequence completion, the model can generate multiple plausible action paths from a single starting point, rather than just the shortest mathematical path.

Conclusion

This work demonstrates that combining mesh-specific spatial encodings with bidirectional temporal modeling provides a powerful tool for 3D animation synthesis. While currently limited to fixed-topology meshes, the framework sets a new standard for stability and realism in mesh-based sequence generation. Future work may explore integration with Generative Adversarial Networks (GANs) to further enhance the diversity and detail of synthesized motions.

Summary updated

Abstract

In this paper, we present a novel method for learning to synthesize 3D mesh animation sequences with long short-term memory (LSTM) blocks and mesh-based convolutional neural networks (CNNs). Synthesizing realistic 3D mesh animation sequences is a challenging and important task in computer animation. To achieve this, researchers have long been focusing on shape analysis to develop new interpolation and extrapolation techniques. However, such techniques have limited learning capabilities and therefore can produce unrealistic animation. Deep architectures that operate directly on mesh sequences remain unexplored, due to the following major barriers: meshes with irregular triangles, sequences containing rich temporal information and flexible deformations. To address these, we utilize convolutional neural networks defined on triangular meshes along with a shape deformation representation to extract useful features, followed by LSTM cells that iteratively process the features. To allow completion of a missing mesh sequence from given endpoints, we propose a new weight-shared bidirectional structure. The bidirectional generation loss also helps mitigate error accumulation over iterations. Benefiting from all these technical advances, our approach outperforms existing methods in sequence prediction and completion both qualitatively and quantitatively. Moreover, this network can also generate follow-up frames conditioned on initial shapes and improve the accuracy as more bootstrap models are provided, which other works in the geometry processing domain cannot achieve.
BibTeX
@misc{qiao2018learning,
  author = {Yi-Ling Qiao and Lin Gao and Yu-Kun Lai and Shihong Xia},
  title = {{Learning Bidirectional LSTM Networks for Synthesizing 3D Mesh Animation Sequences}},
  year = {2018},
  eprint = {1810.02042},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1810.02042},
}
Figure