Deep Sketch-Based Modeling of Man-Made Shapes

Dmitriy Smirnov, Mikhail Bessmeltsev, Justin Solomon

Presented at ICLR 2021, this research introduces a novel approach to 3D shape reconstruction that bridges the gap between deep learning and traditional engineering workflows. By representing man-made objects as collections of parametric patches rather than messy point clouds or dense meshes, the authors enable neural networks to produce 3D models that are clean, compact, and ready for use in professional CAD software.

The Challenge of Man-Made Geometry

Most state-of-the-art 3D deep learning systems output geometry as voxels, point clouds, or dense triangle meshes. While these are effective for visualization, they are difficult to use in industrial design and engineering. Man-made objects, such as cars, planes, or furniture, are typically "piecewise-smooth," consisting of smooth surfaces meeting at sharp edges.

Standard AI outputs are often noisy, lack sharp features, and are incredibly difficult to edit. For instance, tweaking the curve of a car's trunk in a 100,000-point cloud is nearly impossible for a designer. This paper aims to produce parsimonious (efficient) and interpretable 3D outputs that fit directly into the NURBS-based pipelines used in modern CAD software.

Methodology: Deformable Parametric Templates

The core of the method is the Deformable Parametric Template. Instead of generating geometry from scratch, the network learns to deform a pre-defined template of Coons patches to match an input image or sketch.

  1. Representation: The system uses Coons patches, defined by four boundary curves. These are mathematically compatible with NURBS, the industry standard for 3D modeling.
  2. Template Construction: For different categories (e.g., "airplane"), templates are algorithmically generated by simplifying a collection of cuboids into a manifold quad mesh.
  3. Differentiable Losses: To train the network, the authors developed specialized loss functions, including an Area-weighted Chamfer Distance (using the Jacobian of the patch mapping to ensure sampling fairness), Intersection Regularization to prevent patch clipping, and Patch Flatness to encourage sharp creases to align with patch boundaries.
Figure 1: A category-specific template for an airplane, constructed from simplified cuboids.

Key Insights

  • CAD-Native AI: By outputting a small number of patches (e.g., 54 for a sphere template) instead of thousands of triangles, the model becomes inherently more useful for downstream engineering tasks.
  • Semantic Consistency: Because the network deforms a fixed template, the resulting patches are semantically consistent across different models of the same category. A "wing" patch on one airplane will have the same index as the "wing" patch on another, facilitating batch editing and interpolation.
  • Jacobian Weighting: The use of the Jacobian to weight the loss is a crucial technical detail. It ensures that sampling in the $(u, v)$ parameter space translates to uniform accuracy across the actual 3D surface.

Results and Editability

The proposed representation is significantly more compact than existing methods. For a car model, the system achieves comparable accuracy to Pixel2Mesh using only 816 degrees of freedom, compared to the 7,398 required by Pixel2Mesh.

Beyond efficiency, the primary advantage is editability. Because the output is a collection of parametric patches, a user can take the automatically computed model and refine it in standard CAD software by moving just a few control points.

Figure 2: The full pipeline from 2D input sketch to an automatically computed 3D model, which remains natively editable for final refinement.

Limitations and Future Work

The current system relies on a fixed topology; a template for a four-legged chair cannot easily reconstruct a three-legged stool. Furthermore, while the system works with generic templates, the best results require category-specific ones. Future research may involve "learning" the topology itself, allowing the network to automatically add or remove patches to handle more diverse shapes.

Summary updated

Abstract

Sketch-based modeling aims to model 3D geometry using a concise and easy to create---but extremely ambiguous---input: artist sketches. Most conventional sketch-based modeling systems target smooth shapes and, to counter the ambiguity, put manually-designed priors on the 3D shape; they also typically require clean, vectorized input. Recent approaches attempt to learn those priors from data but often produce low-quality output. Focusing on piecewise-smooth man-made shapes, we address these issues by presenting a deep learning-based system to infer a complete man-made 3D shape from a single bitmap sketch. Given a sketch, our system infers a set of parametric surfaces that realize the drawing in 3D. To capture the piecewise smooth geometry of man-made shapes, we learn a special shape representation---a deformable parametric template composed of Coons patches. Naively training such a system, however, would suffer from lack of data and from self-intersections of the parametric surfaces. To address this, we introduce a synthetic sketch augmentation pipeline as well as a loss function that biases the network to output non-self-intersecting shapes. We demonstrate the efficacy of our system on a gallery of synthetic and real artist sketches as well as via comparison to related work.
BibTeX
@inproceedings{smirnov2021deep,
  author = {Dmitriy Smirnov and Mikhail Bessmeltsev and Justin Solomon},
  title = {{Deep Sketch-Based Modeling of Man-Made Shapes}},
  booktitle = {International Conference on Learning Representations},
  year = {2021},
  url = {https://openreview.net/forum?id=jcXYe_1HVg},
}

Loading related content...

Figure