Sketch2CAD: sequential CAD modeling by sketching in context

Changjian Li, Hao Pan, Adrien Bousseau, Niloy J. Mitra

Abstract

Sketch2CAD bridges the gap between the fluidity of freehand sketching and the precision of parametric CAD modeling. By interpreting 2D strokes in the context of an evolving 3D model, the system automatically converts a sequence of sketches into editable, professional-grade CAD operations like extrusions and bevels.

The Design Bottleneck

Industrial design typically involves two disconnected stages: initial ideation through loose freehand sketching and subsequent reconstruction in complex CAD software for manufacturing. This transition is a major bottleneck because existing sketch-to-3D methods often produce "dumb" geometry—point clouds or meshes—that lack the editable design history (the "recipe" of operations) essential for professional workflows.

Sketch2CAD addresses this by performing program synthesis, turning a drawing into a sequence of executable geometric commands. The primary technical challenge is ambiguity; a 2D line can represent an edge, a silhouette, or a cut depending on its placement.

Overview of the Sketch2CAD pipeline, showing the transition from an inspirational sketch to inferred CAD instructions and the final model.

Methods: Context-Aware Interpretation

The core innovation of Sketch2CAD is its use of context-aware interpretation. Instead of analyzing a sketch in isolation, the system processes the sketch alongside the current state of the 3D model using three 256x256 maps: the user's sketch, a depth map, and a normal map.

Two-Stage Deep Learning Pipeline

  1. Classification: A Convolutional Neural Network (CNN) identifies the intended CAD operation: Extrude, Bevel, Add/Subtract Polyhedron, or Sweep.
  2. Segmentation: A U-Net architecture segments the sketch and the model's faces. For instance, in a "Bevel" operation, it identifies which 3D edge is being rounded and which sketched lines represent the new profile.

Parameter Fitting and Regularization

Rather than predicting exact dimensions via regression, the system uses segmentation masks to drive geometric optimization. It fits parametric curves (like Beziers) to the strokes and uses line searches to find precise extrusion depths. To ensure professional quality, the system automatically "cleans up" input by snapping lines to be parallel or perpendicular and regularizing polygons.

Key Insights: Sequentiality and Segmentation

A major "aha moment" in the research is that treating modeling as a conversation—where each sketch builds on the last—simplifies the problem into constrained tasks. This sequential approach allows the system to resolve ambiguity; a circle drawn on a cube is interpreted as a hole or a peg based on its intersection with the existing 3D geometry.

A detailed look at the sequential sketching steps used to build a complex model.

Furthermore, the authors found that segmentation is more robust than regression. By having the neural network label pixels rather than predict floating-point numbers for parameters like "length," traditional geometry solvers can achieve much higher precision in the final CAD output.

Results and Performance

Sketch2CAD demonstrates high accuracy and real-time performance:

  • Accuracy: The system achieved a 91.78% IoU (Intersection over Union) for face detection in the "Extrude" operation.
  • Speed: Neural network inference takes approximately 0.07 seconds, enabling fluid, interactive modeling.
  • User Success: In studies, novices with no prior CAD experience successfully recreated complex objects—such as cameras and desk lamps—within 5 to 20 minutes.

Limitations and Future Work

Currently, the system is limited to four basic operations and is optimized for planar faces. It does not yet handle complex features like lofts, shells, or patterns, nor does it possess "semantic awareness" (e.g., knowing it is building a "wheel" versus a generic cylinder). Future iterations aim to incorporate semantic priors and support sketching on highly curved surfaces.

Summary updated

BibTeX
@article{li2020sketch,
  author = {Changjian Li and Hao Pan and Adrien Bousseau and Niloy J. Mitra},
  title = {{Sketch2CAD: sequential CAD modeling by sketching in context}},
  journal = {ACM Trans. Graphics},
  year = {2020},
  doi = {10.1145/3414685.3417807},
}
Figure