TL;DR
TapMo is a text-to-animation pipeline capable of animating virtually any 3D character—even those without predefined skeletons—using natural language descriptions. By combining an adaptive mesh handle predictor with a shape-aware diffusion model, it generates realistic, mesh-specific motions for diverse characters like jellyfish, ghosts, and monsters, significantly outperforming traditional human-centric animation methods.
Introduction & Problem Statement
Most state-of-the-art 3D motion generation models are inherently "human-locked." They rely heavily on the SMPL model, a parametric human template with a fixed skeletal structure. While this approach excels at generating standard human locomotion, it fails completely when applied to "skeleton-free" or non-humanoid characters, such as a multi-limbed alien, a blob, or inanimate objects.
To animate these unconventional characters today, creators usually resort to manual skeletal rigging—a notoriously tedious process—or auto-rigging tools that often yield inconsistent results across different shapes. The core challenge this paper addresses is creating a universal motion representation that perceives a character's unique geometry and generates motion that does not distort or "break" the underlying mesh.
Methodology
TapMo (Text-driven Animation Pipeline for Motion) overcomes these limitations by introducing two synergistic components:
1. Mesh Handle Predictor
Instead of relying on a predefined skeleton, this module utilizes a Graph Convolutional Network (GCN) to analyze the input mesh. It automatically predicts "handles" (clusters of vertices that act as control points) alongside "skinning weights" (which dictate how much each vertex moves in relation to a handle). This innovation allows TapMo to establish a unified motion representation across vastly heterogeneous characters, treating a humanoid hand and a jellyfish tentacle using the same mathematical framework.
2. Shape-aware Motion Diffusion
TapMo employs a generative model based on the Diffusion framework. It takes two primary inputs: a text prompt (encoded via CLIP) and a low-dimensional feature representing the character's shape.
Unlike standard models that output rigid joint rotations, TapMo generates a mesh-specific adaptation ($\delta$). This set of local translations fine-tunes the motion to the specific geometry of the character. By feeding the mesh's shape features back into the Diffusion model, the system "knows" whether it is animating a wide character versus a thin one, adjusting limb trajectories accordingly to prevent the mesh from stretching or collapsing unnaturally.
Weakly-Supervised Training
Because massive datasets of "text-to-jellyfish" motions do not exist, the authors trained the model using human motion data paired with clever auxiliary losses:
- Spring Loss ($L_h$): Treats the character's limbs like a spring system to ensure "bone" lengths remain consistent during movement.
- Adversarial Loss ($L_a$): Uses a discriminator to ensure the handle-driven mesh motion looks as natural as traditional skeleton-driven motion.
- ARAP Objective: The authors integrated "As-Rigid-As-Possible" (ARAP) principles into the fine-tuning stage, ensuring that the local surface of the 3D model remains undistorted even during complex movements.
Datasets
The system was trained and evaluated using four major public datasets:
- AMASS: A large-scale database of human motion.
- Mixamo: A repository of diverse 3D virtual characters with various skeletons.
- ModelsResource-RigNet: A dataset containing 2,703 rigged characters with high shape variety.
- HumanML3D: A dataset of 3D human motions paired with textual descriptions, crucial for training the text-to-motion capabilities.
Key Results
TapMo demonstrates substantial improvements over existing auto-animation baselines:
- Motion Quality: TapMo achieved a 50% improvement in FID (Frechet Inception Distance) compared to a standard MDM implementation, indicating significantly more realistic and diverse generated motions.
- Geometry Preservation: In quantitative tests for mesh distortion (ARAP-Loss), TapMo scored roughly 70% lower than baseline methods (MDM-SR and MDM-SFR). This translates to far fewer "glitches" or structural collapses in the 3D mesh during animation.
- User Preference: In a perceptual study with 100 volunteers, TapMo was preferred over baseline methods more than 84% of the time across categories evaluating text-matching and geometry quality.
Limitations and Future Work
While highly effective, TapMo has a few limitations. First, it exhibits rest pose sensitivity. The model assumes characters start in a standard "T-pose." If a character is provided in a non-standard rest pose (e.g., already sitting or crouching), the model can suffer from mesh breakage. Second, due to a human-centric bias in the training data, the model struggles to generate strictly non-humanoid locomotion, such as a four-legged gallop or a bird flapping its wings.
For future work, the authors suggest exploring a "unified generation" approach—simultaneously generating both the 3D mesh and its corresponding motion directly from a single text prompt.
Summary updated