Learning Skeletal Articulations with Neural Blend Shapes

Peizhuo Li, Kfir Aberman, Rana Hanocka, Libin Liu, Olga Sorkine-Hornung, Baoquan Chen

TL;DR

This paper introduces a neural framework that automatically rigs and skins 3D character meshes of any topology, producing high-quality animations compatible with standard motion capture (mocap) data. By combining a base skeletal deformation model with "neural blend shapes"—learned corrective displacements—the system eliminates common artifacts like "elbow collapse" without requiring ground-truth rigging data for training.

What Problem Does This Paper Address?

In the world of 3D animation, bringing a static character mesh to life is a grueling process. Animators must first "rig" the character (create a skeleton) and then "skin" it (define how much each bone influences specific parts of the mesh). Standard techniques like Linear Blend Skinning (LBS) are computationally efficient but prone to ugly artifacts, such as joints losing volume when bent (the "elbow collapse" or "candy wrapper" effect).

While recent deep learning methods have tried to automate this, they often struggle with two things:

  1. Connectivity Constraints: Many models only work on meshes with a fixed number of vertices and specific edge connections.
  2. Data Scarcity: High-quality rigging and skinning data is hard to come by. Most existing models require direct supervision from artist-created rigs, which are expensive to produce.

This paper aims to create a system that works on arbitrary meshes, produces mocap-compatible rigs, and learns how to deform them simply by looking at the final moving shapes, rather than the underlying rig parameters.

Methods

The authors propose a two-branch neural architecture that mimics the classical animation pipeline:

  1. Envelope Deformation Branch: This branch handles the "base" movement. It uses MeshCNN to extract features from the input mesh surface and a skeleton-aware network to predict joint offsets (the rig) and skinning weights. Crucially, the skeletal hierarchy is pre-defined to match common mocap formats, ensuring the output is immediately usable in game engines.
  2. Residual Deformation Branch: This is the "secret sauce." It predicts neural blend shapes—pose-dependent displacements that correct the errors of the base envelope. As the character moves, a small MLP calculates blending coefficients based on joint rotations, adding fine-grained detail and volume preservation to joint regions.
Method overview showing the two-branch architecture: the Envelope branch for base rigging and the Residual branch for corrective blend shapes.

A standout feature is Indirect Supervision. Instead of being told "this vertex should have a weight of 0.5 for the arm bone," the network is trained by comparing its final articulated mesh to a ground-truth deformed shape. It "back-calculates" the best rigging and skinning parameters to achieve that final look.

Datasets

The model was trained and evaluated using a mix of synthetic and real-world data:

  • SMPL Dataset: A standard parametric human body model used to generate thousands of diverse body shapes and poses.
  • Multi-Garment Network (MGN): Used to provide data on clothed humans, helping the network learn to handle geometric variability like loose shirts or pants.
  • Mixamo: A library of professional character rigs used to test the model's ability to generalize to "in the wild" characters created by 3D artists.
  • Mesh Augmentation: The authors applied topological operations (edge flips, splits, collapses) to the training meshes to ensure the network remains robust regardless of how a mesh is triangulated.

Key Insights and Ideas

  • Rigging as Attention-Based Pooling: The authors treat the rigging process (finding joint positions) as a form of attention-based pooling. Joint positions are inferred by looking at the deep features of the mesh vertices that are most strongly "bound" to them via skinning weights.
  • The Power of Indirect Supervision: By supervising only the final vertex positions, the network is free to find the most efficient rigging and skinning parameters for a given topology, often outperforming methods trained on direct artist supervision.
  • Generalization through Augmentation: Training on "broken" or augmented meshes (edge collapses/splits) is what allows the network to handle a high-resolution professional character just as easily as a low-poly game asset.

Key Results

The framework demonstrates significant improvements over state-of-the-art methods like RigNet and Pinocchio:

  • Rigging Accuracy: In quantitative tests (measuring distances between predicted and ground-truth joints), this method achieved a Chamfer Distance (CD-J2J) of 0.012, compared to RigNet's 0.194 and Pinocchio's 0.474.
  • Deformation Quality: The neural blend shapes effectively eliminate volume loss. As seen in Figure 1, the joints look natural even in extreme poses.
  • Plug-and-Play: Because the output conforms to standard LBS/DQS models with corrective blend shapes, the rigs can be imported directly into software like Maya, Blender, or Unreal Engine.
Fig. 1. Our neural skeletal articulation network learns to rig and skin an input character with arbitrary connectivity, and generates neural blend shapes. Our framework produces pose-dependent displacements that result in high quality deformations, especially in the joint regions.

Limitations and Future Work

  • Fixed Hierarchy: The current model requires a pre-defined skeletal structure (e.g., a standard biped). To rig a quadruped or a creature with extra limbs, the network would need to be retrained.
  • Static Deformations: The model focuses on pose-dependent corrections but does not yet account for secondary dynamics—the "jiggle" of muscles or the sway of clothing that happens due to momentum.
  • Input Requirements: The system assumes the input character is in a T-pose and has a consistent upright orientation. Relaxing these requirements is a goal for future research.

Summary updated

Abstract

Animating a newly designed character using motion capture (mocap) data is a long standing problem in computer animation. A key consideration is the skeletal structure that should correspond to the available mocap data, and the shape deformation in the joint regions, which often requires a tailored, pose-specific refinement. In this work, we develop a neural technique for articulating 3D characters using enveloping with a pre-defined skeletal structure which produces high quality pose dependent deformations. Our framework learns to rig and skin characters with the same articulation structure (e.g., bipeds or quadrupeds), and builds the desired skeleton hierarchy into the network architecture. Furthermore, we propose neural blend shapes--a set of corrective pose-dependent shapes which improve the deformation quality in the joint regions in order to address the notorious artifacts resulting from standard rigging and skinning. Our system estimates neural blend shapes for input meshes with arbitrary connectivity, as well as weighting coefficients which are conditioned on the input joint rotations. Unlike recent deep learning techniques which supervise the network with ground-truth rigging and skinning parameters, our approach does not assume that the training data has a specific underlying deformation model. Instead, during training, the network observes deformed shapes and learns to infer the corresponding rig, skin and blend shapes using indirect supervision. During inference, we demonstrate that our network generalizes to unseen characters with arbitrary mesh connectivity, including unrigged characters built by 3D artists. Conforming to standard skeletal animation models enables direct plug-and-play in standard animation software, as well as game engines.
BibTeX
@misc{li2021learning,
  author = {Peizhuo Li and Kfir Aberman and Rana Hanocka and Libin Liu and Olga Sorkine-Hornung and Baoquan Chen},
  title = {{Learning Skeletal Articulations with Neural Blend Shapes}},
  year = {2021},
  eprint = {2105.02451},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2105.02451},
}

Loading related content...

Figure