Abstract
Deep3DSketch is a deep learning framework designed to reconstruct high-fidelity 3D meshes from single, free-hand sketches. By disentangling object shape from camera viewpoint and employing a structural-aware adversarial training process, the framework addresses the inherent ambiguity and sparsity of sketch-based 3D modeling. Experimental results on ShapeNet benchmarks demonstrate that Deep3DSketch achieves state-of-the-art performance, particularly in capturing fine-grained structural details that previous methods often fail to recover.
Introduction
While professional 3D modeling typically requires mastery of complex CAD software, sketching remains the most natural and intuitive way for humans to express geometric ideas. However, converting a 2D sketch into a 3D model is a mathematically ill-posed problem. Sketches are inherently sparse, lacking the texture and depth cues found in photographs, and are often ambiguous—a single set of lines can represent multiple potential 3D volumes.
Existing AI models often struggle with these challenges, frequently producing "blobby" results or failing to capture thin, fine-grained structures like chair legs or table supports. Deep3DSketch addresses these limitations by introducing a framework that explicitly interprets the 3D depth hidden behind 2D human strokes through view-aware disentanglement and adversarial learning.
Methodology
Deep3DSketch utilizes an encoder-decoder architecture enhanced by several novel components designed to resolve structural ambiguity:
1. View-Aware Disentanglement
Rather than attempting to map a sketch directly to a 3D volume, the model explicitly predicts the viewpoint (Euler angles) of the input sketch. By separating the object's identity ("what" it is) from its orientation ("where" it is being viewed), the decoder can focus on generating a consistent 3D structure in a canonical coordinate space. This "un-rotates" the problem, significantly simplifying the reconstruction task.
2. Random Pose Sampling (RPS)
To ensure the generated model is truly three-dimensional and not just a 2.5D projection, the system employs Random Pose Sampling. During training, the system generates the mesh and then "observes" it from multiple random viewpoints. This forces the model to maintain structural integrity from angles not visible in the original sketch.
3. Progressive Structural-Aware Discriminator
The framework incorporates a Generative Adversarial Network (GAN) approach. A discriminator network evaluates 2D silhouettes of the generated 3D model from various angles. This discriminator is "progressive," meaning it initially focuses on low-resolution outlines before gradually shifting its attention to high-resolution details. This progression forces the generator to prioritize global shape first and then refine fine-grained structural elements.
4. Differentiable Rendering
To bridge the gap between 3D meshes and 2D silhouettes, the team utilizes SoftRas, a differentiable renderer. This allows gradients from 2D image-based losses to flow back through the pipeline, directly updating the 3D mesh generation process based on how well the 2D projections match the target.
Datasets and Evaluation
The model was rigorously evaluated using two primary datasets:
- ShapeNet-Synthetic: A large-scale dataset where 3D models are converted into clean 2D edge maps using Canny edge detection, providing a baseline for "perfect" sketches.
- ShapeNet-Sketch: A more challenging real-world dataset consisting of 1,300 free-hand drawings created by human volunteers. This dataset tests the model's robustness against the messy, varied strokes typical of non-artists.
Key Results
Deep3DSketch achieved State-of-the-Art (SOTA) performance across 13 object categories, including chairs, tables, and airplanes.
- Quantitative Performance: On the ShapeNet-Synthetic benchmark, the model achieved a mean Voxel IoU of 0.618, outperforming the previous leading method, Sketch2Model (0.601).
- Qualitative Improvements: Visual comparisons highlight the model's superior ability to capture "thin" structures. While previous models often merged four distinct chair legs into a single solid block, Deep3DSketch successfully reconstructs individual legs and the open spaces between them.
- Robustness: The model maintained high fidelity even when processing messy human sketches, demonstrating a successful bridge over the "domain gap" between synthetic lines and hand-drawn strokes.
Conclusion and Future Work
Deep3DSketch represents a significant step forward in making 3D modeling accessible through sketching. By leveraging adversarial training and viewpoint disentanglement, it overcomes the traditional "blobby" output of sketch-to-3D systems.
However, limitations remain. The model currently relies on deforming a "template mesh" (like a sphere), which can restrict its ability to represent complex topologies, such as objects with holes. Additionally, the current iteration requires training on specific object categories. Future research will focus on developing a "universal" model capable of understanding diverse object categories simultaneously and further refining domain adaptation for even more varied human sketching styles.
Summary updated