Abstract
3D modeling based on point clouds is an efficient way to reconstruct and create detailed 3D content. However, the geometric procedure may lose accuracy due to high redundancy and the absence of an explicit structure. In this work, we propose a human-in-the-loop sketch-based point cloud reconstruction framework to leverage users' cognitive abilities in geometry extraction. We present an interactive drawing interface for 3D model creation from point cloud data with the help of user sketches. We adopt an optimization method in which the user can continuously edit the contours extracted from the obtained 3D model and retrieve the model iteratively. Finally, we verify the proposed user interface for modeling from sparse point clouds.
Introduction
Reconstructing 3D shapes from point clouds is a cornerstone of computer vision, but it faces a significant "sparse data" problem. While dense point clouds from high-end LiDAR are relatively easy to mesh, data from consumer-grade depth cameras or binocular setups is often noisy, incomplete, or sparse.
Standard automated tools often struggle with these inputs because they lack an explicit structure to follow, frequently producing distorted or "blobby" results when points are missing. This research argues that while algorithms struggle with ambiguity, humans are excellent at "filling in the blanks" and recognizing 3D structures from minimal visual cues. The challenge lies in creating an interface that allows humans to guide the reconstruction process efficiently.
Methodology: The Human-in-the-Loop Cycle
The authors propose an interactive, eight-step workflow centered on a "human-in-the-loop" cycle that treats the user as a powerful spatial prior.
- Observation & Sketching: The user rotates the sparse point cloud to a favorable angle and draws a rough 2D sketch of the object’s silhouette.
- Sketch-Based Retrieval: The system uses the OpenSSE engine to compare the user's sketch against a pre-computed database of 10,200 2D contours from various 3D models.
- Model Alignment: The retrieved 3D model is aligned with the original sparse point cloud using the Iterative Closest Point (ICP) algorithm to ensure a spatial fit.
- Iterative Refinement: If the retrieved model is not perfect, the system extracts its contours and displays them back to the user. The user can then edit these contours directly—rather than redrawing from scratch—to better match the point cloud.
- Re-retrieval: The edited contours are used to perform a new search, narrowing down the target model until the user is satisfied.
Experimental Results
The system was prototyped using a subset of the SHREC 2012 dataset, containing 100 point clouds across five categories: teacup, chair, table, vase, and animal. A user study with six participants evaluated the system's effectiveness in real-world scenarios.
Key quantitative and qualitative results include:
- Efficiency: On average, users required only 1.67 sketching iterations to find the correct target model.
- Accuracy: The mean ICP error (measuring the fit between the model and the point cloud) was low at 5.37 x 10^-2, indicating stable and accurate alignment.
- User Experience: Participants rated the "Easiness to Use" at 4.33/5 and "User Satisfaction" at 3.83/5.
- Viewpoint Agnostic Modeling: Because users can rotate the point cloud and sketch from any angle, the system is not limited by the original sensor's perspective.
Conclusion and Future Work
The proposed framework demonstrates that human intuition can effectively resolve geometric ambiguities in sparse data reconstruction. By allowing users to tweak extracted contours rather than redrawing them, the system creates a tight and intuitive feedback loop.
Future work aims to expand the dataset variety to include more complex, organic shapes like the human body. Additionally, the authors suggest moving from a retrieval-based engine to a deep learning-based generative model, which could allow the system to synthesize new 3D geometry based on user sketches rather than simply selecting from a pre-existing database.
Summary updated