TapMo: Shape-aware Motion Generation of Skeleton-free Characters

Jiaxu Zhang, Shaoli Huang, Zhigang Tu, Xin Chen, Xiaohang Zhan, Gang Yu, Ying Shan

TL;DR

TapMo is a text-to-animation pipeline capable of animating virtually any 3D character—even those without predefined skeletons—using natural language descriptions. By combining an adaptive mesh handle predictor with a shape-aware diffusion model, it generates realistic, mesh-specific motions for diverse characters like jellyfish, ghosts, and monsters, significantly outperforming traditional human-centric animation methods.

Introduction & Problem Statement

Most state-of-the-art 3D motion generation models are inherently "human-locked." They rely heavily on the SMPL model, a parametric human template with a fixed skeletal structure. While this approach excels at generating standard human locomotion, it fails completely when applied to "skeleton-free" or non-humanoid characters, such as a multi-limbed alien, a blob, or inanimate objects.

Comparison of existing human-locked methods versus TapMo animating diverse skeleton-free characters.

To animate these unconventional characters today, creators usually resort to manual skeletal rigging—a notoriously tedious process—or auto-rigging tools that often yield inconsistent results across different shapes. The core challenge this paper addresses is creating a universal motion representation that perceives a character's unique geometry and generates motion that does not distort or "break" the underlying mesh.

Methodology

TapMo (Text-driven Animation Pipeline for Motion) overcomes these limitations by introducing two synergistic components:

1. Mesh Handle Predictor

Instead of relying on a predefined skeleton, this module utilizes a Graph Convolutional Network (GCN) to analyze the input mesh. It automatically predicts "handles" (clusters of vertices that act as control points) alongside "skinning weights" (which dictate how much each vertex moves in relation to a handle). This innovation allows TapMo to establish a unified motion representation across vastly heterogeneous characters, treating a humanoid hand and a jellyfish tentacle using the same mathematical framework.

2. Shape-aware Motion Diffusion

TapMo employs a generative model based on the Diffusion framework. It takes two primary inputs: a text prompt (encoded via CLIP) and a low-dimensional feature representing the character's shape.

Unlike standard models that output rigid joint rotations, TapMo generates a mesh-specific adaptation ($\delta$). This set of local translations fine-tunes the motion to the specific geometry of the character. By feeding the mesh's shape features back into the Diffusion model, the system "knows" whether it is animating a wide character versus a thin one, adjusting limb trajectories accordingly to prevent the mesh from stretching or collapsing unnaturally.

Weakly-Supervised Training

Because massive datasets of "text-to-jellyfish" motions do not exist, the authors trained the model using human motion data paired with clever auxiliary losses:

  • Spring Loss ($L_h$): Treats the character's limbs like a spring system to ensure "bone" lengths remain consistent during movement.
  • Adversarial Loss ($L_a$): Uses a discriminator to ensure the handle-driven mesh motion looks as natural as traditional skeleton-driven motion.
  • ARAP Objective: The authors integrated "As-Rigid-As-Possible" (ARAP) principles into the fine-tuning stage, ensuring that the local surface of the 3D model remains undistorted even during complex movements.

Datasets

The system was trained and evaluated using four major public datasets:

  • AMASS: A large-scale database of human motion.
  • Mixamo: A repository of diverse 3D virtual characters with various skeletons.
  • ModelsResource-RigNet: A dataset containing 2,703 rigged characters with high shape variety.
  • HumanML3D: A dataset of 3D human motions paired with textual descriptions, crucial for training the text-to-motion capabilities.

Key Results

TapMo demonstrates substantial improvements over existing auto-animation baselines:

  • Motion Quality: TapMo achieved a 50% improvement in FID (Frechet Inception Distance) compared to a standard MDM implementation, indicating significantly more realistic and diverse generated motions.
  • Geometry Preservation: In quantitative tests for mesh distortion (ARAP-Loss), TapMo scored roughly 70% lower than baseline methods (MDM-SR and MDM-SFR). This translates to far fewer "glitches" or structural collapses in the 3D mesh during animation.
  • User Preference: In a perceptual study with 100 volunteers, TapMo was preferred over baseline methods more than 84% of the time across categories evaluating text-matching and geometry quality.

Limitations and Future Work

While highly effective, TapMo has a few limitations. First, it exhibits rest pose sensitivity. The model assumes characters start in a standard "T-pose." If a character is provided in a non-standard rest pose (e.g., already sitting or crouching), the model can suffer from mesh breakage. Second, due to a human-centric bias in the training data, the model struggles to generate strictly non-humanoid locomotion, such as a four-legged gallop or a bird flapping its wings.

For future work, the authors suggest exploring a "unified generation" approach—simultaneously generating both the 3D mesh and its corresponding motion directly from a single text prompt.

Summary updated

Abstract

Previous motion generation methods are limited to the pre-rigged 3D human model, hindering their applications in the animation of various non-rigged characters. In this work, we present TapMo, a Text-driven Animation PIpeline for synthesizing Motion in a broad spectrum of skeleton-free 3D characters. The pivotal innovation in TapMo is its use of shape deformation-aware features as a condition to guide the diffusion model, thereby enabling the generation of mesh-specific motions for various characters. Specifically, TapMo comprises two main components - Mesh Handle Predictor and Shape-aware Diffusion Module. Mesh Handle Predictor predicts the skinning weights and clusters mesh vertices into adaptive handles for deformation control, which eliminates the need for traditional skeletal rigging. Shape-aware Motion Diffusion synthesizes motion with mesh-specific adaptations. This module employs text-guided motions and mesh features extracted during the first stage, preserving the geometric integrity of the animations by accounting for the character's shape and deformation. Trained in a weakly-supervised manner, TapMo can accommodate a multitude of non-human meshes, both with and without associated text motions. We demonstrate the effectiveness and generalizability of TapMo through rigorous qualitative and quantitative experiments. Our results reveal that TapMo consistently outperforms existing auto-animation methods, delivering superior-quality animations for both seen or unseen heterogeneous 3D characters.
BibTeX
@inproceedings{zhang2024tapmo,
  author = {Jiaxu Zhang and Shaoli Huang and Zhigang Tu and Xin Chen and Xiaohang Zhan and Gang YU and Ying Shan},
  title = {{TapMo: Shape-aware Motion Generation of Skeleton-free Characters}},
  booktitle = {International Conference on Learning Representations},
  year = {2024},
  url = {https://openreview.net/forum?id=OeH6Fdhv7q},
}
Figure