SketchBodyNet: A Sketch-Driven Multi-faceted Decoder Network for 3D Human Reconstruction

Fei Wang, Kongzhang Tang, Hefeng Wu, Baoquan Zhao, Hao Cai, Teng ZhouarXiv 2023 · DOI · arXiv

Introduction

3D human reconstruction has advanced significantly using natural images, but freehand sketches present a unique challenge. Sketches are abstract, often anatomically imprecise, and lack the rich visual cues like texture and shading found in photographs. This makes the mapping from 2D lines to 3D meshes a "badly ill-posed" problem. Traditional models often struggle with sketches because they attempt to predict camera, shape, and pose parameters through a single shared decoder, leading to "mutual interference" where errors in one facet degrade the others.

Example of a sparse freehand sketch input, illustrating the lack of detail that makes 3D reconstruction challenging.

The SketchBodyNet Architecture

To address these challenges, the authors propose SketchBodyNet, a deep learning architecture specifically optimized for the sparse and noisy nature of hand-drawn lines. The network follows a three-stage pipeline:

  1. Feature Extraction: A ResNet50 backbone serves as the encoder, extracting high-level features from the input sketch into a 2048-dimensional vector.
  2. Multi-faceted Attention Decoders: The core innovation is the separation of the decoding process into three distinct branches: CamDecoder, ShapeDecoder, and PoseDecoder. Each branch utilizes a multi-head self-attention module. This allows the network to focus on specific "key positions" or strokes relevant to each parameter—for instance, focusing on joint intersections for pose while considering overall proportions for shape.
  3. Mesh Generation: The predicted shape ($\beta$) and pose ($\theta$) parameters are integrated into the Skinned Multi-Person Linear (SMPL) model. The camera parameter ($cam$) ensures the resulting 3D mesh aligns correctly with the 2D sketch plane.

Training Strategy and Datasets

The researchers introduced Sketch3D, a large-scale benchmark containing approximately 26,000 freehand sketches paired with 3D ground truth meshes. These sketches were collected from 41 volunteers drawing humans in diverse poses from 14 different viewpoints.

To bridge the gap between perfect geometry and human drawing variability, the team employed Step-wise Refinement Training (SRT). The model is first pre-trained on "synthetic sketches" (edge maps generated from 3D projections) to learn basic structural relationships. It is then fine-tuned on the Sketch3D freehand dataset to adapt to the nuances of human artistic style.

Experimental Results

SketchBodyNet demonstrates superior performance over existing state-of-the-art methods like HMR, SPIN, and HybrIK.

  • Quantitative Performance: The model achieved a Mean Per Joint Position Error (MPJPE) of 154.89, a significant improvement over the previous best-performing model, MAED (176.79).
  • Alignment Accuracy: It reached an image-model alignment accuracy of 86.93%, outperforming all baseline methods.
  • Qualitative Quality: Visual results show that SketchBodyNet effectively handles complex poses, such as crouching or twisting, where other models often fail to resolve limb positions correctly.
Qualitative reconstruction results showing input sketches, their alignment with the reconstructed mesh, and the final 3D model from different angles.

Conclusion

By decoupling the prediction of camera, shape, and pose into multi-faceted attention-based decoders, SketchBodyNet successfully mitigates the interference issues common in 3D reconstruction from sparse inputs. While challenges remain regarding extreme anatomical distortions in very poor drawings, the model sets a new standard for sketch-based human modeling, opening doors for intuitive 3D design tools.

Summary updated

Abstract

Reconstructing 3D human shapes from 2D images has received increasing attention recently due to its fundamental support for many high-level 3D applications. Compared with natural images, freehand sketches are much more flexible to depict various shapes, providing a high potential and valuable way for 3D human reconstruction. However, such a task is highly challenging. The sparse abstract characteristics of sketches add severe difficulties, such as arbitrariness, inaccuracy, and lacking image details, to the already badly ill-posed problem of 2D-to-3D reconstruction. Although current methods have achieved great success in reconstructing 3D human bodies from a single-view image, they do not work well on freehand sketches. In this paper, we propose a novel sketch-driven multi-faceted decoder network termed SketchBodyNet to address this task. Specifically, the network consists of a backbone and three separate attention decoder branches, where a multi-head self-attention module is exploited in each decoder to obtain enhanced features, followed by a multi-layer perceptron. The multi-faceted decoders aim to predict the camera, shape, and pose parameters, respectively, which are then associated with the SMPL model to reconstruct the corresponding 3D human mesh. In learning, existing 3D meshes are projected via the camera parameters into 2D synthetic sketches with joints, which are combined with the freehand sketches to optimize the model. To verify our method, we collect a large-scale dataset of about 26k freehand sketches and their corresponding 3D meshes containing various poses of human bodies from 14 different angles. Extensive experimental results demonstrate our SketchBodyNet achieves superior performance in reconstructing 3D human meshes from freehand sketches.
BibTeX
@misc{wang2023sketchbodynet,
  author = {Fei Wang and Kongzhang Tang and Hefeng Wu and Baoquan Zhao and Hao Cai and Teng Zhou},
  title = {{SketchBodyNet: A Sketch-Driven Multi-faceted Decoder Network for 3D Human Reconstruction}},
  year = {2023},
  eprint = {2310.06577},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2310.06577},
}
Figure