TL;DR
This paper introduces the first automatically generated dataset of 2D line-labeled contours with convexity information, derived from 3D models. This novel dataset enables training of segmentation models for contour detection and classification, significantly improving downstream tasks like 3D reconstruction.
What Problem Does This Paper Address?
Understanding the fine details of 3D objects, particularly their contours and how they bend (convexity), is crucial for many computer vision applications, such as robotic manipulation and 3D shape recognition. However, existing datasets lack this detailed line labeling, and manual annotation is prohibitively time-consuming and expensive. This gap hinders the development of robust models that can accurately interpret object shapes from 2D images.
Methods
The authors propose an automated pipeline to generate a 2D line-labeled dataset from 3D models. The process involves three main stages:
- Contour Extraction: 3D contours are extracted from 3D STL meshes by identifying intersections of adjacent triangles. Surface normal discontinuities are used to filter out non-contour edges, and a ray-casting method handles occlusions to produce final 2D contours.
- Contour Classification: Contours are classified into three types: concave, convex, and obscuring. This is achieved by analyzing triangle visibility and applying an Extended Convexity Criterion based on vectors to triangle centroids and unit normals.
- Contour Grouping: Small, linear contour segments are grouped into smoother curves. This is done by constructing a weighted graph where contours are nodes and junctions are edges, then finding a minimum spanning tree to link segments based on local curvature and proximity.
Datasets
- Thingi10K dataset: The authors utilize the genus 0 and genus 1 subsets of the Thingi10K dataset, which contains a variety of 3D printed models.
- Novel 2D Line-Labeled Dataset: The paper introduces a new dataset consisting of 6,275 automatically generated 2D line-labeled instances with convexity labels. This dataset is derived from the Thingi10K models.
Key Insights and Ideas
- Automated Labeling is Feasible: The paper demonstrates that it's possible to automatically generate high-quality line-labeled datasets from 3D models, bypassing the need for manual annotation. This opens the door to creating much larger and more diverse datasets in the future.
- Convexity is Key for Understanding: Explicitly labeling contour convexity provides valuable geometric information that significantly aids downstream tasks, particularly 3D reconstruction, by reducing ambiguity.
- Generalizability of Segmentation Models: The authors show that state-of-the-art segmentation models can be trained on this novel dataset to achieve good performance, even on unseen objects, highlighting the dataset's potential for generalization.
Key Results
- Dataset Creation: The pipeline successfully generates 6,275 labeled 2D instances with contour extraction, classification, and grouping. Error metrics for each stage are reported to be low (e.g., extraction error < 1%, labeling error < 2%, grouping error < 5%).
- Segmentation Performance: State-of-the-art models like Mask2former and SOLOv2 achieve promising results on the dataset. Mask2former with early fusion of Bézier curve data shows the best performance, with mAP scores for seen objects reaching up to 29.37% and for unseen objects up to 23.1%. Contour Accuracy (CA) for Mask2former+earlyfusion is reported as 0.81 for seen and 0.74 for unseen objects.
- 3D Reconstruction Improvement: Integrating the contour labels into the 3DR2N2 3D reconstruction framework significantly improves performance. IoU scores increase across various input sizes and voxel resolutions, with an improvement from 0.45 to 0.53 for seen objects and 0.23 to 0.31 for unseen objects when using 256x256 input and 32x32x32 voxels.
Limitations and Future Work
- Dataset Size: While 6,275 instances are generated, the authors acknowledge that larger datasets would further improve generalization.
- Generalization to Real-World Images: The current dataset is generated from rendered 3D models. Future work could focus on adapting these models to perform well on real-world images with varying lighting, textures, and noise.
- More Complex Models: The current dataset is derived from genus 0 and 1 subsets of Thingi10K. Expanding to more complex shapes and categories would be beneficial.
- Downstream Tasks: The authors suggest exploring other downstream tasks beyond 3D reconstruction, such as robotic grasping and object affordance prediction, where contour information is critical.
Summary updated