NeuralAnnot: Neural Annotator for 3D Human Mesh Training Sets

Gyeongsik Moon, Hongsuk Choi, Kyoung Mu LeeWorkshops @ CVPR 2022 · OpenReview

Abstract

Most 3D human mesh regressors are fully supervised with 3D pseudo-GT human model parameters and weakly supervised with GT 2D/3D joint coordinates as the 3D pseudo-GTs bring great performance gain. The 3D pseudo-GTs are obtained by annotators, systems that iteratively fit 3D human model parameters to GT 2D/3D joint coordinates of training sets in the pre-processing stage of the regressors. Optimization-based annotators, such as SMPLify-X, have been widely used but often produce wrong 3D pseudo-GTs as they fit the 3D parameters to each sample independently. To overcome this, we present NeuralAnnot, a neural network-based annotator. Assuming no 3D pseudo-GTs are available, NeuralAnnot is weakly supervised with GT 2D/3D joint coordinates. The testing results on the same training sets become 3D pseudo-GTs, used to fully supervise regressors. We show that 3D pseudo-GTs of NeuralAnnot are highly beneficial to train the regressors.

Introduction

Reconstructing 3D human meshes from 2D images is a fundamental task in computer vision, yet it suffers from a significant data scarcity problem. While Motion Capture (MoCap) datasets provide precise 3D ground truth (GT), they lack the visual diversity of real-world "in-the-wild" images. Conversely, in-the-wild datasets lack 3D labels. To bridge this gap, researchers employ "annotators"—optimization-based systems like SMPLify-X that fit 3D models to 2D joint coordinates. However, these methods often fail on complex poses or occlusions because they treat each image in isolation, lacking a global understanding of human anatomy.

NeuralAnnot: A Data-Driven Annotator

NeuralAnnot addresses these limitations by replacing iterative optimization with a "one-stage" neural network. Instead of solving a per-image math problem, NeuralAnnot learns a data-driven prior by being trained on the target dataset itself.

NeuralAnnot workflow for MoCap and in-the-wild datasets.

Methodology

NeuralAnnot utilizes a ResNet-50 backbone followed by a Multi-Layer Perceptron (MLP) to regress 3D parameters. The training process follows a unique cycle:

  1. Weak Supervision: The network is trained using only available 2D or 3D joint coordinates.
  2. Pseudo-GT Generation: Once trained, the network is run on the same training set. Its predictions serve as high-quality 3D pseudo-GTs.
  3. Downstream Training: These pseudo-GTs are then used to provide full supervision for final human mesh regressors.

By processing the image as input, NeuralAnnot can resolve depth ambiguities using visual cues like shadows or relative limb sizes—information that purely coordinate-based optimization methods ignore.

Evaluation Metrics

The authors evaluate the quality of the generated annotations using two distinct approaches: measuring how accurately the pseudo-GTs match the actual 3D ground truth (Direct Error), and measuring the performance improvement of a regressor when trained using these pseudo-GTs (Indirect Error).

Evaluation metrics: Direct vs Indirect 3D annotation error.

Results and Discussion

NeuralAnnot demonstrates significant improvements across multiple benchmarks, including Human3.6M, 3DPW, and MSCOCO.

Quantitative Performance

On the 3DPW dataset, NeuralAnnot achieved a Direct Error (MPJPE) of 10.7mm, nearly doubling the accuracy of the optimization-based SMPLify-X (19.2mm). Furthermore, regressors trained on NeuralAnnot’s pseudo-GTs consistently outperformed those trained on labels from previous state-of-the-art methods like SPIN or EFT. For instance, on MSCOCO, the indirect error (PA MPVPE) was reduced from 80.7mm to 71.6mm.

Qualitative Robustness

A key advantage of NeuralAnnot is its robustness to image truncation and occlusion. While optimization methods often produce anatomically impossible "spaghetti" limbs when a person is partially cut off, NeuralAnnot uses its learned knowledge of human proportions to realistically "hallucinate" missing parts. It also shows superior performance in crowded scenes and complex hand-object interactions.

Conclusion

NeuralAnnot provides a powerful framework for generating high-quality 3D annotations for diverse datasets. By shifting from per-sample optimization to a data-driven neural approach, it overcomes long-standing issues of depth ambiguity and anatomical inconsistency. The public release of these 3D pseudo-GTs offers a valuable resource for the community to train more accurate and robust human mesh regressors.

Summary updated

Abstract

Most 3D human mesh regressors are fully supervised with 3D pseudo-GT human model parameters and weakly supervised with GT 2D/3D joint coordinates as the 3D pseudo-GTs bring great performance gain. The 3D pseudo-GTs are obtained by annotators, systems that iteratively fit 3D human model parameters to GT 2D/3D joint coordinates of training sets in the pre-processing stage of the regressors. The fitted 3D parameters at the last fitting iteration become the 3D pseudo-GTs, used to fully super-vise the regressors. Optimization-based annotators, such as SMPLify-X, have been widely used to obtain the 3D pseudo-GTs. However, they often produce wrong 3D pseudo-GTs as they fit the 3D parameters to GT of each sample independently. To overcome the limitation, we present NeuralAnnot, a neural network-based annotator. The main idea of NeuralAnnot is to employ a neural network-based regressor and dedicate it for the annotation. Assuming no 3D pseudo-GTs are available, NeuralAnnot is weakly supervised with GT 2D/3D joint coordinates of training sets. The testing results on the same training sets become 3D pseudo-GTs, used to fully supervise the regressors. We show that 3D pseudo-GTs of NeuralAnnot are highly beneficial to train the regressors. We made our 3D pseudo-GTs publicly available.
BibTeX
@inproceedings{moon2022neuralannot,
  author = {Gyeongsik Moon and Hongsuk Choi and Kyoung Mu Lee},
  title = {{NeuralAnnot: Neural Annotator for 3D Human Mesh Training Sets}},
  booktitle = {Workshops @ CVPR},
  year = {2022},
  url = {https://openreview.net/forum?id=fwMMGSacio},
}
Figure