Introduction
Estimating a 3D human mesh from a single 2D image is a fundamental task in computer vision with broad applications in AR/VR and biomechanics. A primary hurdle in this field is the scarcity of "paired" datasets—images with corresponding ground-truth 3D labels. While motion capture (MoCap) systems provide high-fidelity 3D data, they typically lack associated RGB images. Conversely, in-the-wild images often have 2D keypoint annotations but lack 3D information.
Previous methods attempting to bridge this gap often struggle with the "domain gap": 2D keypoints projected from clean MoCap data differ significantly from the noisy detections produced by real-world estimators like OpenPose. Key2Mesh addresses this by utilizing a fast, non-iterative Multi-Layer Perceptron (MLP) architecture and an adversarial domain adaptation strategy.
Methodology
Key2Mesh employs a two-stage training pipeline designed to learn robust 3D reconstructions from unpaired data.
Stage 1: Pre-Training (Source Domain)
The model is initially trained on the AMASS dataset, which contains extensive 3D human motions. 3D meshes are projected into 2D keypoints via virtual cameras. To prepare the model for real-world noise, heavy data augmentation—including random rotations, occlusions, and jitter—is applied. The model learns to regress SMPL (Skinned Multi-Person Linear model) parameters directly from these synthetic 2D points.
Stage 2: Adversarial Domain Adaptation (Target Domain)
To close the gap between synthetic and real-world keypoints, the authors introduce a Domain Critic. This sub-network is trained to distinguish between features extracted from "clean" MoCap-derived keypoints and "noisy" keypoints detected on real images (e.g., from the InstaVariety or Human3.6M datasets).
By training the feature extractor to "fool" the Domain Critic, the system learns domain-invariant representations. This process requires no 3D labels for the visual target domain, relying instead on the alignment of feature distributions.
Experimental Results
Key2Mesh demonstrates a superior balance of accuracy and efficiency compared to existing state-of-the-art (SOTA) methods.
- State-of-the-Art Accuracy: On the 3DPW outdoor benchmark, Key2Mesh achieves a PA-MPJPE of 49.8 mm, outperforming complex iterative models like LGD and STRAPS. Ablation studies confirm that the domain adaptation stage is critical, reducing error on the Human3.6M dataset from 53.9 mm to 51.0 mm.
- Massive Speedup: The non-iterative MLP architecture allows for extremely fast inference. Key2Mesh processes a single frame in just 3.4 ms, making it 12x faster than LGD for individual samples and up to 33x faster when processing large batches.
Discussion and Limitations
Beyond performance, Key2Mesh offers a strategic advantage for privacy-sensitive applications. By using 2D keypoints as a proxy and discarding raw RGB pixels during the mesh estimation phase, the model minimizes the risk associated with storing identifiable image data.
However, the model faces certain challenges. The reliance on 2D keypoints can lead to depth ambiguity (e.g., determining the relative position of limbs to the torso). Furthermore, performance is inherently limited by the accuracy of the underlying 2D pose estimator. Future work may involve integrating temporal information across frame sequences to improve consistency and resolve ambiguous poses.
Summary updated