Abstract
Parametric models of humans, faces, hands and animals have been widely used for a range of tasks such as image-based reconstruction, shape correspondence estimation, and animation. Their key strength is the ability to factor surface variations into shape and pose dependent components. Learning such models requires lots of expert knowledge and hand-defined object-specific constraints, making the learning approach unscalable to novel objects. In this paper, we present a simple yet effective approach to learn disentangled shape and pose representations in an unsupervised setting. We use a combination of self-consistency and cross-consistency constraints to learn pose and shape space from registered meshes. We additionally incorporate as-rigid-as-possible deformation (ARAP) into the training loop to avoid degenerate solutions. We demonstrate the usefulness of learned representations through a number of tasks including pose transfer and shape retrieval. The experiments on datasets of 3D humans, faces, hands and animals demonstrate the generality of our approach.
Introduction
A fundamental challenge in 3D computer vision is teaching machines to distinguish between an object's intrinsic shape (its identity) and its current pose (its motion or expression). For instance, a tall person waving and a short person waving share a pose but differ in shape. Conversely, a tall person sitting and a tall person standing share a shape but differ in pose.
Traditional parametric models, such as SMPL for humans, rely on extensive expert knowledge, including predefined skeleton hierarchies and manual skinning weights. These requirements make it difficult to scale such models to new categories like specific animal species or artistic styles. This paper introduces an unsupervised auto-encoder that achieves this disentanglement without skeletons, blend weights, or manual annotations.
Methodology: ARAP-in-the-Loop
The authors utilize a mesh auto-encoder with two independent encoder branches: one for shape ($\beta$) and one for pose ($\theta$). To force these branches to specialize, the framework employs two primary constraints:
- Cross-consistency: By swapping shape codes between two meshes of the same subject in different poses, the model is forced to reconstruct the target mesh accurately.
- As-Rigid-As-Possible (ARAP) Consistency: To address the lack of "same pose, different shape" pairs in real-world data, the authors generate "proxy" meshes. A hybrid mesh is created using the pose from one subject and the shape from another. This hybrid is then passed through an ARAP optimization loop. ARAP is a geometric technique that deforms a mesh while preserving local surface details, ensuring the pose branch cannot "cheat" by sneaking shape information into its latent code.
The architecture leverages Spiral Convolutions, allowing the network to process 3D mesh surfaces directly rather than relying on inefficient voxel grids or point clouds.
Experiments and Datasets
The model's generality was validated across four distinct datasets:
- AMASS: Human motion data.
- SMAL: Quadruped animals (lions, tigers, horses).
- MANO: Human hands.
- COMA: Human facial expressions.
All datasets utilize "registered" meshes, where every mesh in a category shares the same vertex count and topology.
Key Results
- Pose Transfer: The model can accurately apply the pose of one subject to the shape of another. On the AMASS dataset, it achieved a vertex-to-vertex error of 19.43mm, a significant improvement over the previous unsupervised state-of-the-art, GDVAE (54.44mm).
- Near-Supervised Accuracy: The unsupervised approach performs remarkably well, coming within 4mm of the accuracy achieved by fully supervised models trained with ground-truth labels.
- Shape Retrieval: The disentangled shape code allows for high-accuracy subject retrieval in databases, regardless of the subject's pose.
Limitations and Future Directions
While effective, the method currently requires registered meshes, meaning it cannot yet process raw scans with varying topologies. Additionally, because the pose space is learned unsupervised, the latent variables lack semantic meaning (e.g., there is no specific "elbow rotation" parameter). Future work will focus on extending these techniques to un-registered point clouds to increase flexibility for real-world sensor data.
Summary updated