Three-dimensional reconstruction of industrial parts from a single image

Zhenxing Xu, Aizeng Wang, Fei Hou, Gang ZhaoVisual 2024 · DOI

TL;DR

Researchers have developed a deep learning pipeline that reconstructs high-fidelity 3D CAD models from a single 2D image of an industrial part. By predicting specific geometric parameters and poses rather than raw point clouds, the system generates industry-standard NURBS (Non-Uniform Rational B-splines) surfaces with sub-millimeter accuracy.

What Problem Does This Paper Address?

In the world of intelligent manufacturing, robots often need to "see" a part and immediately understand its 3D structure to perform tasks like automated loading or quality inspection. Traditional 3D reconstruction usually relies on multi-view geometry (needing many images) or 3D scanning, which produces "point clouds."

Point clouds are messy; they are collections of coordinates that lack the mathematical "cleanliness" required for Computer-Aided Design (CAD). Industrial parts are almost universally defined by NURBS, a mathematical representation that is smooth, scalable, and precise. The challenge is bridging the gap between a flat, 2D photograph and a parametric, vector-based 3D model in real-time.

Methods

The authors propose a two-stage deep learning approach to extract the "DNA" of a part from an image:

  1. CAD-ClassNet: A convolutional neural network (CNN) that identifies the category of the part (e.g., is this a bolt or a gear?).
  2. CAD-ReconNet: Once the type is known, this second network extracts feature vectors representing the part's pose and dimensions.
  3. Feature Comparison & Interpolation: Instead of just "guessing" the size, the system compares the extracted features against a Standard Feature Library using cosine similarity. It finds the two closest matches in the library and uses linear interpolation to calculate the exact dimensions (like the modulus of a gear or the length of a bolt).
  4. NURBS Generation: These predicted parameters are plugged into a geometric modeling engine to generate the final 3D vector model.

This differs from standard approaches because it doesn't try to "sculpt" a 3D shape from pixels; it identifies the part and then "tunes" a pre-defined mathematical template to match the image.

Datasets

Because no comprehensive dataset existed for this specific task, the authors built their own:

  • Industrial Part Dataset: Contains 16,800 synthetic 2D images generated from CAD models.
  • Scope: Five typical industrial categories: hexagonal head bolts, cylindrical gears, shoulder rings, hexagonal nuts, and cylindrical roller bearings.
  • Variability: Each part type includes 10 different sizes and 336 different shooting poses (angles) to ensure the model is robust to perspective changes.
  • Real-world Testing: The authors also captured images of physical parts in natural scenes to verify the model's performance outside of a simulated environment.

Key Insights and Ideas

  • Parameters over Pixels: The core "aha" moment is that industrial parts are highly standardized. You don't need to reconstruct every vertex; you just need to find the five or six numbers (parameters) that define that specific part.
  • NURBS as the Output: By targeting NURBS rather than voxels or meshes, the output is immediately usable in manufacturing software without further processing.
  • The Symmetry Challenge: The researchers found that highly symmetrical parts (like hexagonal nuts) are actually harder for the AI to reconstruct accurately because small changes in size or orientation look nearly identical from many angles.
  • Efficiency: By using a "classification" mindset for pose and a "similarity" mindset for dimensions, the system avoids the massive computational overhead of traditional 3D computer vision.

Key Results

  • Classification Accuracy: The CAD-ClassNet achieved 100% accuracy in identifying the part type across the test set.
  • Reconstruction Precision: The accuracy of the final 3D shape reconstruction was less than 0.1 mm in error.
  • Speed: On a mid-range workstation (NVIDIA RTX 3070), the entire process—from image input to 3D model—takes roughly 19–20 seconds.
  • Robustness: When compared against standard architectures like ResNet-18 and ResNet-34, the custom CAD-ReconNet showed better convergence and higher testing accuracy for industrial features.

Limitations and Future Work

While the results are promising, the authors note a few hurdles:

  • Symmetry Issues: As mentioned, the hexagonal nut had a lower testing accuracy (~81.85%) compared to other parts due to its structural symmetry.
  • Preprocessing Requirements: For real-world images, the system currently requires background culling and regularization to "clean up" the image before the networks can process it.
  • Complexity: The current model is trained on five specific part types. Expanding this to the thousands of unique components found in a typical factory would require a much larger feature library and more complex classification hierarchies.

Future research will likely focus on automating the background removal process and expanding the dataset to include more complex, non-standardized industrial geometries.

Summary updated

Abstract

This study proposes an image-based three-dimensional (3D) vector reconstruction of industrial parts that can generate non-uniform rational B-splines (NURBS) surfaces with high fidelity and flexibility. The contributions of this study include three parts: first, a dataset of two-dimensional images is constructed for typical industrial parts, including hexagonal head bolts, cylindrical gears, shoulder rings, hexagonal nuts, and cylindrical roller bearings; second, a deep learning algorithm is developed for parameter extraction of 3D industrial parts, which can determine the final 3D parameters and pose information of the reconstructed model using two new nets, CAD-ClassNet and CAD-ReconNet; and finally, a 3D vector shape reconstruction of mechanical parts is presented to generate NURBS from the obtained shape parameters. The final reconstructed models show that the proposed approach is highly accurate, efficient, and practical.
BibTeX
@inproceedings{xu2024three,
  author = {Zhenxing Xu and Aizeng Wang and Fei Hou and Gang Zhao},
  title = {{Three-dimensional reconstruction of industrial parts from a single image}},
  booktitle = {Visual},
  year = {2024},
  doi = {10.1186/s42492-024-00158-7},
}
Figure