TapMo: Shape-aware Motion Generation of Skeleton-free Characters

Jiaxu Zhang, Shaoli Huang, Zhigang Tu, Xin Chen, Xiaohang Zhan, Gang Yu, Ying Shan

Abstract

Previous motion generation methods are limited to the pre-rigged 3D human model, hindering their applications in the animation of various non-rigged characters. In this work, we present TapMo, a Text-driven Animation Pipeline for synthesizing Motion in a broad spectrum of skeleton-free 3D characters. The pivotal innovation in TapMo is its use of shape deformation-aware features as a condition to guide the diffusion model, thereby enabling the generation of mesh-specific motions for various characters. Specifically, TapMo comprises two main components - Mesh Handle Predictor and Shape-aware Diffusion Module. Mesh Handle Predictor predicts the skinning weights and clusters mesh vertices into adaptive handles for deformation control, which eliminates the need for traditional skeletal rigging. Shape-aware Motion Diffusion synthesizes motion with mesh-specific adaptations. This module employs text-guided motions and mesh features extracted during the first stage, preserving the geometric integrity of the animations by accounting for the character's shape and deformation. Trained in a weakly-supervised manner, TapMo can accommodate a multitude of non-human meshes, both with and without associated text motions. We demonstrate the effectiveness and generalizability of TapMo through rigorous qualitative and quantitative experiments. Our results reveal that TapMo consistently outperforms existing auto-animation methods, delivering superior-quality animations for both seen or unseen heterogeneous 3D characters.

Introduction

Traditional 3D animation is a bottlenecked process. Most state-of-the-art motion generation models, such as MDM or TEMOS, are "human-centric," relying heavily on the SMPL model—a fixed human skeletal structure. While this works for humanoid avatars, it fails for the diverse "heterogeneous" characters found in games and VR, such as multi-legged aliens or sentient everyday objects. Existing "auto-rigging" solutions attempt to fit skeletons into these shapes, but they often produce inconsistent results or require manual cleanup. Furthermore, standard motion models do not "perceive" the character's geometry; they might generate a "walk" that causes a wide-bodied character's mesh to tear or its limbs to clip through its torso. TapMo aims to bridge this gap by creating a unified, skeleton-free representation that is aware of a character's specific shape.

Comparison of TapMo with existing human-centric motion generation methods.

Methodology

TapMo introduces a two-stage pipeline designed to automate the journey from a static mesh to a moving character without the need for traditional skeletal rigging.

Mesh Handle Predictor

Instead of a rigid bone hierarchy, TapMo uses a Graph Convolutional Network (GCN) to predict adaptive handles. It clusters mesh vertices into semantic parts (e.g., identifying a group of vertices as a limb) and assigns skinning weights. Crucially, it predicts a "root handle" to manage global movement, allowing the model to handle characters with wildly different topologies, from jellyfish to teapots.

Shape-aware Motion Diffusion

This module acts as the system's "brain," utilizing a Transformer-based Diffusion Model. It processes three primary inputs: noised motion, text prompts (via CLIP), and mesh deformation features extracted in the first stage.

  • Motion Adaptation ($\delta$): The model does not just output a generic motion; it generates a mesh-specific "offset" ($\delta$) to fine-tune the movement for that specific body shape. This ensures that the character's geometry is a first-class citizen in the diffusion process, preventing mesh collapse or distortion.
The TapMo pipeline showing mesh handle prediction and resulting motion sequences.

Training and Datasets

Since there is no massive dataset of non-humanoid characters performing specific actions, TapMo is trained in a weakly-supervised manner. The authors leveraged major benchmarks including HumanML3D, AMASS, Mixamo, and ModelsResource-RigNet. To ensure natural movement, clever geometric constraints were added:

  • Spring Loss: Treats limbs like a spring system to ensure "bone" lengths stay consistent during movement.
  • Adversarial Loss: A discriminator ensures the generated mesh movement looks as natural as traditional skeleton-driven animation.

Key Results

TapMo significantly outperforms existing baselines in both motion accuracy and geometric integrity:

  • Quantitative Performance: On the HumanML3D benchmark, TapMo achieved an FID of 0.515, a massive improvement over the 1.058 achieved by standard diffusion implementations.
  • Geometry Quality: TapMo reduced mesh distortion (measured by ARAP-Loss) by 60-70% compared to baseline retargeting methods.
  • User Preference: In a user study with 100 volunteers, over 84% of users preferred TapMo’s animations, citing better alignment with text and fewer visual glitches.

Limitations and Future Work

While TapMo is a leap forward, some hurdles remain. The model currently assumes characters are in a standard "T-pose"; non-standard rest poses can still cause mesh breakage. Additionally, because it is trained primarily on human motion data, it is currently better at anthropomorphic movements than quadrupedal gallops or entirely alien locomotion. Future work aims to integrate TapMo with text-to-3D mesh generation models for a truly end-to-end animation pipeline.

Summary updated

Abstract

Previous motion generation methods are limited to the pre-rigged 3D human model, hindering their applications in the animation of various non-rigged characters. In this work, we present TapMo, a Text-driven Animation Pipeline for synthesizing Motion in a broad spectrum of skeleton-free 3D characters. The pivotal innovation in TapMo is its use of shape deformation-aware features as a condition to guide the diffusion model, thereby enabling the generation of mesh-specific motions for various characters. Specifically, TapMo comprises two main components - Mesh Handle Predictor and Shape-aware Diffusion Module. Mesh Handle Predictor predicts the skinning weights and clusters mesh vertices into adaptive handles for deformation control, which eliminates the need for traditional skeletal rigging. Shape-aware Motion Diffusion synthesizes motion with mesh-specific adaptations. This module employs text-guided motions and mesh features extracted during the first stage, preserving the geometric integrity of the animations by accounting for the character's shape and deformation. Trained in a weakly-supervised manner, TapMo can accommodate a multitude of non-human meshes, both with and without associated text motions. We demonstrate the effectiveness and generalizability of TapMo through rigorous qualitative and quantitative experiments. Our results reveal that TapMo consistently outperforms existing auto-animation methods, delivering superior-quality animations for both seen or unseen heterogeneous 3D characters.
BibTeX
@inproceedings{zhang2024tapmo,
  author = {Jiaxu Zhang and Shaoli Huang and Zhigang Tu and Xin Chen and Xiaohang Zhan and Gang Yu and Ying Shan},
  title = {{TapMo: Shape-aware Motion Generation of Skeleton-free Characters}},
  booktitle = {International Conference on Learning Representations},
  year = {2024},
  url = {https://openreview.net/forum?id=OeH6Fdhv7q},
}
Figure