Abstract
We propose Seg&Struct, a supervised learning framework leveraging the interplay between part segmentation and structure inference and demonstrating their synergy in an integrated framework. Both part segmentation and structure inference have been extensively studied in the recent deep learning literature, while the supervisions used for each task have not been fully exploited to assist the other task. Namely, structure inference has been typically conducted with an autoencoder that does not leverage the point-to-part associations. Also, segmentation has been mostly performed without structural priors that tell the plausibility of the output segments. We present how these two tasks can be best combined while fully utilizing supervision to improve performance. Our framework first decomposes a raw input shape into part segments using an off-the-shelf algorithm, whose outputs are then mapped to nodes in a part hierarchy, establishing point-to-part associations. Following this, ours predicts the structural information, e.g., part bounding boxes and part relationships. Lastly, the segmentation is rectified by examining the confusion of part boundaries using the structure-based part features. Our experimental results based on the StructureNet and PartNet demonstrate that the interplay between two tasks results in remarkable improvements in both tasks: 27.91% in structure inference and 0.5% in segmentation.
Introduction
Parsing a 3D shape—understanding that a "chair" is composed of a "back," "seat," and "legs" arranged in a specific way—is a fundamental challenge in computer vision. Traditionally, researchers have tackled this in two isolated ways: part segmentation (labeling every point in a 3D point cloud) and structure inference (predicting a hierarchy of bounding boxes and relationships).
The primary limitation of this isolation is that segmentation often lacks "global common sense," leading to noisy boundaries, while structure inference often ignores the actual input geometry, leading to "hallucinated" structures that do not match the specific object. Seg&Struct proposes that these two tasks should operate in a bidirectional loop to resolve these discrepancies.
Methodology
The authors propose a two-stage "interplay" framework that links local geometry with global structural priors.
1. Segmentation-to-Structure (The Forward Path)
Instead of encoding a whole shape into a single abstract vector, the model first runs an off-the-shelf segmenter to decompose the shape into raw parts. These parts are treated as "leaf nodes" in a hierarchy. By using skip connections to link these specific input segments directly to the structure decoder, the model ensures the predicted bounding boxes and hierarchy are "grounded" in the actual points of the input shape.
2. Structure-to-Segmentation Refinement (The Backward Path)
The model then uses the predicted structure to rectify the initial segmentation. It identifies "conflicts" where predicted bounding boxes overlap. A refinement network examines these overlapping areas; if the global structure suggests these parts should be unified (e.g., a seat accidentally split into two), it merges them. This uses structural context to solve local ambiguities that a point-level segmenter would miss.
Experimental Results
The model was trained and evaluated on the PartNet and StructureNet benchmarks, focusing on complex, multi-part categories: Chairs, Tables, and Storage furniture.
Quantitative and Qualitative Performance
- Structure Inference: The model achieved a 27.91% improvement in Average Precision (AP) for part prediction compared to baselines that did not use the segmentation-to-structure interplay.
- Segmentation Refinement: The refinement module improved segmentation accuracy by 0.5%. While numerically small, qualitative results (Figure 4) show this specifically fixes "logical" errors, such as incorrectly split parts, leading to much cleaner and more consistent outputs.
- Structural Retrieval: The authors demonstrate that the predicted structure is accurate enough to be used for shape retrieval, aligning better with human perception of "logical" similarity than standard geometric metrics.
Conclusion
Seg&Struct demonstrates that closing the loop between part segmentation and structure inference significantly enhances 3D shape parsing. By grounding structural predictions in geometry and using those structures to refine local segments, the model achieves a more robust understanding of 3D objects. Future work may address the current limitation of the refinement module, which focuses on merging over-segmented parts but does not yet handle under-segmentation (splitting parts).
Summary updated