TL;DR
This paper introduces StPNet, a deep learning framework that allows users to create 3D surfaces interactively from sparse 3D sketch strokes. It tackles the challenge of inferring smooth, plausible surfaces from incomplete and irregular sketch data, enabling real-time 3D model editing and in-painting.
What Problem Does This Paper Address?
Creating 3D models is often a complex and time-consuming process, especially for artists accustomed to the intuitive nature of 2D sketching. While digital tools have enabled 3D modeling, the interaction mechanisms can be difficult to learn. This paper addresses the gap in creating a fluid, sketch-based workflow for 3D surface generation and editing. Specifically, it focuses on translating sparse, irregularly placed 3D sketch strokes into coherent 3D surfaces, a problem closely related to scattered data approximation. Existing methods often struggle with the sparsity and irregularity of sketch data, leading to sub-optimal surface fitting. The paper also highlights the challenge of "surface in-painting," where missing parts of a 3D model need to be filled in, and traditional methods offer limited creative control.
Methods
The core of the proposed solution is StPNet, a deep neural network designed for incremental surface fitting from a stream of 3D sketch strokes. StPNet comprises two main modules:
- Global Surface Estimate: This module takes the incoming 3D sketch strokes and produces an initial, coarse approximation of the 3D surface. This step helps to translate the irregular input into a more structured representation.
- Local Surface Estimate: This module refines the surface prediction by incorporating local features and considering the history of strokes. It uses an LSTM layer to capture the temporal evolution of the sketch, allowing for incremental updates and a more accurate fit to the evolving sketch.
The network outputs control points for a bicubic Bézier surface, which offers a good balance between processing ease and representational versatility. For interactive editing, the method can also be used for surface in-painting, where new strokes are used to fill in missing or existing patches in a 3D model. The system is integrated as a plugin for Blender, a popular 3D content creation tool.
Datasets
- ABCSpline dataset: Derived from the ABC dataset, this dataset consists of B-spline surfaces converted to bicubic Bézier surfaces, with random strokes sampled from them.
- boundABCSpline dataset: Similar to ABCSpline, but includes boundary strokes for the surfaces.
- ABCSplineClosed and boundABCSplineClosed datasets: These datasets are created from closed surfaces, split into open patches for testing the method's ability to handle closed objects.
- RBST (boundRBST) dataset: A synthetic dataset of randomly generated 3D bicubic Bézier surfaces used to test the model's generalization capabilities.
- Utah tea set models: Used for evaluating the interactive surface creation and in-painting capabilities on real-world 3D models.
Key Insights and Ideas
- Incremental Learning for Sketching: The paper demonstrates the effectiveness of an incremental, stream-based approach for fitting surfaces to 3D sketches. This mirrors the natural way an artist sketches and allows for real-time feedback.
- Global-Local Feature Fusion: Combining a global surface estimation with a local refinement module proves crucial for handling sparse and irregular sketch data, capturing both overall shape and fine details.
- Bézier Surfaces for Flexibility: Using bicubic Bézier surfaces as the output representation provides a good trade-off between the complexity of the surface and the ease of computation and manipulation.
- Creative Control in In-painting: The method offers artists more direct creative input in the surface in-painting process compared to traditional methods that rely on post-hoc adjustments.
- Boundary Strokes are Informative: Explicitly sketching boundary strokes significantly improves the approximation accuracy, highlighting their importance in defining the patch's context.
Key Results
- Quantitative Improvement: StPNet consistently outperforms several state-of-the-art baselines (e.g., Parsenet, Foldingnet) in terms of surface fitting error across various datasets. For instance, it achieves a 91.46% improvement over the best baseline on the boundABCSpline dataset.
- Stroke Efficiency: The method shows strong performance even with a small number of input strokes, demonstrating its ability to infer plausible surfaces from sparse data.
- Interactive Performance: The Blender plugin demonstrates real-time in-painting and editing capabilities, allowing users to interactively create and refine 3D surfaces.
- Generalization: The model shows good generalization to unseen datasets (RBST) and can approximate closed surfaces without explicit training on them.
Limitations and Future Work
- Dependency on Boundary Strokes: While boundary strokes improve performance, the paper notes that future work could aim to reduce the dependency on explicitly defining them, enabling the network to adapt to its neighborhood automatically.
- Organic Shapes: The current work focuses on patch-based models. Future directions include extending the method to handle more complex organic shapes, such as human faces.
- Creation from Scratch: The authors express interest in exploring the use of their network for creating 3D models entirely from scratch, beyond just in-painting or patch fitting.
Summary updated