Abstract
Most 3D human mesh regressors are fully supervised with 3D pseudo-GT human model parameters and weakly supervised with GT 2D/3D joint coordinates as the 3D pseudo-GTs bring great performance gain. The 3D pseudo-GTs are obtained by annotators, systems that iteratively fit 3D human model parameters to GT 2D/3D joint coordinates of training sets in the pre-processing stage of the regressors. Optimization-based annotators, such as SMPLify-X, have been widely used but often produce wrong 3D pseudo-GTs as they fit the 3D parameters to each sample independently. To overcome this, we present NeuralAnnot, a neural network-based annotator. Assuming no 3D pseudo-GTs are available, NeuralAnnot is weakly supervised with GT 2D/3D joint coordinates. The testing results on the same training sets become 3D pseudo-GTs, used to fully supervise regressors. We show that 3D pseudo-GTs of NeuralAnnot are highly beneficial to train the regressors.
Introduction
Reconstructing 3D human meshes from 2D images is a fundamental task in computer vision, yet it suffers from a significant data scarcity problem. While Motion Capture (MoCap) datasets provide precise 3D ground truth (GT), they lack the visual diversity of real-world "in-the-wild" images. Conversely, in-the-wild datasets lack 3D labels. To bridge this gap, researchers employ "annotators"—optimization-based systems like SMPLify-X that fit 3D models to 2D joint coordinates. However, these methods often fail on complex poses or occlusions because they treat each image in isolation, lacking a global understanding of human anatomy.
NeuralAnnot: A Data-Driven Annotator
NeuralAnnot addresses these limitations by replacing iterative optimization with a "one-stage" neural network. Instead of solving a per-image math problem, NeuralAnnot learns a data-driven prior by being trained on the target dataset itself.
Methodology
NeuralAnnot utilizes a ResNet-50 backbone followed by a Multi-Layer Perceptron (MLP) to regress 3D parameters. The training process follows a unique cycle:
- Weak Supervision: The network is trained using only available 2D or 3D joint coordinates.
- Pseudo-GT Generation: Once trained, the network is run on the same training set. Its predictions serve as high-quality 3D pseudo-GTs.
- Downstream Training: These pseudo-GTs are then used to provide full supervision for final human mesh regressors.
By processing the image as input, NeuralAnnot can resolve depth ambiguities using visual cues like shadows or relative limb sizes—information that purely coordinate-based optimization methods ignore.
Evaluation Metrics
The authors evaluate the quality of the generated annotations using two distinct approaches: measuring how accurately the pseudo-GTs match the actual 3D ground truth (Direct Error), and measuring the performance improvement of a regressor when trained using these pseudo-GTs (Indirect Error).
Results and Discussion
NeuralAnnot demonstrates significant improvements across multiple benchmarks, including Human3.6M, 3DPW, and MSCOCO.
Quantitative Performance
On the 3DPW dataset, NeuralAnnot achieved a Direct Error (MPJPE) of 10.7mm, nearly doubling the accuracy of the optimization-based SMPLify-X (19.2mm). Furthermore, regressors trained on NeuralAnnot’s pseudo-GTs consistently outperformed those trained on labels from previous state-of-the-art methods like SPIN or EFT. For instance, on MSCOCO, the indirect error (PA MPVPE) was reduced from 80.7mm to 71.6mm.
Qualitative Robustness
A key advantage of NeuralAnnot is its robustness to image truncation and occlusion. While optimization methods often produce anatomically impossible "spaghetti" limbs when a person is partially cut off, NeuralAnnot uses its learned knowledge of human proportions to realistically "hallucinate" missing parts. It also shows superior performance in crowded scenes and complex hand-object interactions.
Conclusion
NeuralAnnot provides a powerful framework for generating high-quality 3D annotations for diverse datasets. By shifting from per-sample optimization to a data-driven neural approach, it overcomes long-standing issues of depth ambiguity and anatomical inconsistency. The public release of these 3D pseudo-GTs offers a valuable resource for the community to train more accurate and robust human mesh regressors.
Summary updated