Introduction
3D human reconstruction has advanced significantly using natural images, but freehand sketches present a unique challenge. Sketches are abstract, often anatomically imprecise, and lack the rich visual cues like texture and shading found in photographs. This makes the mapping from 2D lines to 3D meshes a "badly ill-posed" problem. Traditional models often struggle with sketches because they attempt to predict camera, shape, and pose parameters through a single shared decoder, leading to "mutual interference" where errors in one facet degrade the others.
The SketchBodyNet Architecture
To address these challenges, the authors propose SketchBodyNet, a deep learning architecture specifically optimized for the sparse and noisy nature of hand-drawn lines. The network follows a three-stage pipeline:
- Feature Extraction: A ResNet50 backbone serves as the encoder, extracting high-level features from the input sketch into a 2048-dimensional vector.
- Multi-faceted Attention Decoders: The core innovation is the separation of the decoding process into three distinct branches: CamDecoder, ShapeDecoder, and PoseDecoder. Each branch utilizes a multi-head self-attention module. This allows the network to focus on specific "key positions" or strokes relevant to each parameter—for instance, focusing on joint intersections for pose while considering overall proportions for shape.
- Mesh Generation: The predicted shape ($\beta$) and pose ($\theta$) parameters are integrated into the Skinned Multi-Person Linear (SMPL) model. The camera parameter ($cam$) ensures the resulting 3D mesh aligns correctly with the 2D sketch plane.
Training Strategy and Datasets
The researchers introduced Sketch3D, a large-scale benchmark containing approximately 26,000 freehand sketches paired with 3D ground truth meshes. These sketches were collected from 41 volunteers drawing humans in diverse poses from 14 different viewpoints.
To bridge the gap between perfect geometry and human drawing variability, the team employed Step-wise Refinement Training (SRT). The model is first pre-trained on "synthetic sketches" (edge maps generated from 3D projections) to learn basic structural relationships. It is then fine-tuned on the Sketch3D freehand dataset to adapt to the nuances of human artistic style.
Experimental Results
SketchBodyNet demonstrates superior performance over existing state-of-the-art methods like HMR, SPIN, and HybrIK.
- Quantitative Performance: The model achieved a Mean Per Joint Position Error (MPJPE) of 154.89, a significant improvement over the previous best-performing model, MAED (176.79).
- Alignment Accuracy: It reached an image-model alignment accuracy of 86.93%, outperforming all baseline methods.
- Qualitative Quality: Visual results show that SketchBodyNet effectively handles complex poses, such as crouching or twisting, where other models often fail to resolve limb positions correctly.
Conclusion
By decoupling the prediction of camera, shape, and pose into multi-faceted attention-based decoders, SketchBodyNet successfully mitigates the interference issues common in 3D reconstruction from sparse inputs. While challenges remain regarding extreme anatomical distortions in very poor drawings, the model sets a new standard for sketch-based human modeling, opening doors for intuitive 3D design tools.
Summary updated