Abstract
We propose Seg&Struct, a supervised learning framework leveraging the interplay between part segmentation and structure inference and demonstrating their synergy in an integrated framework. Both part segmentation and structure inference have been extensively studied in the recent deep learning literature, while the supervisions used for each task have not been fully exploited to assist the other task. Namely, structure inference has been typically conducted with an autoencoder that does not leverage the point-to-part associations. Also, segmentation has been mostly performed without structural priors that tell the plausibility of the output segments. We present how these two tasks can be best combined while fully utilizing supervision to improve performance. Our framework first decomposes a raw input shape into part segments using an off-the-shelf algorithm, whose outputs are then mapped to nodes in a part hierarchy, establishing point-to-part associations. Following this, ours predicts the structural information, e.g., part bounding boxes and part relationships. Lastly, the segmentation is rectified by examining the confusion of part boundaries using the structure-based part features. Our experimental results based on the StructureNet and PartNet demonstrate that the interplay between the two tasks results in remarkable improvements in both tasks: 27.91% in structure inference and 0.5% in segmentation.
Introduction
Parsing a 3D shape into its constituent parts—such as the legs, seat, and back of a chair—is a fundamental task in computer vision. Traditionally, researchers have approached this from two distinct angles: part segmentation and structure inference. Part segmentation involves labeling every point in a 3D point cloud, which provides local detail but often results in "noisy" outputs that lack global plausibility (e.g., a chair leg split into disjoint pieces). Conversely, structure inference predicts the hierarchy and relationships (like symmetry or adjacency) between parts. While this captures the "big picture," it often ignores the underlying geometry, leading to generic structures that fail to match the specific input shape.
Seg&Struct addresses this gap by treating these tasks as two sides of the same coin. The authors argue that segmentation requires structural priors to ensure plausibility, while structure inference needs point-to-part associations to remain grounded in the actual geometry of the object.
Methodology: The Bidirectional Loop
The core of the Seg&Struct framework is a bidirectional loop that facilitates a synergy between geometry and structure. This process is divided into two main phases: the forward path (Segmentation-to-Structure) and the backward path (Structure-to-Segmentation Refinement).
Segmentation-to-Structure (The Forward Path)
The system begins by using an off-the-shelf parser, such as PointGroup, to decompose the raw 3D shape into initial segments. These segments are treated as "leaf nodes" in a hierarchical tree. The framework builds this hierarchy bottom-up, aggregating geometric features into a "root" code. A decoder then works top-down to predict part bounding boxes and relationships. Crucially, the model employs skip connections from the original segments, ensuring the predicted structure reflects the specific input geometry rather than a generic template.
Structure-to-Segmentation Refinement (The Backward Path)
In the refinement phase, the model utilizes the predicted structure to identify and resolve geometric conflicts, such as overlapping bounding boxes or messy boundaries. A refinement network (denoted as $\mathcal{M}$) predicts "merge" operations to clean the initial segmentation. For instance, if the structural model determines that two segments should functionally be a single part—like a table leg accidentally split during initial parsing—it merges them to produce a more coherent result.
Experimental Evaluation
The framework was evaluated using two major 3D datasets: PartNet, for training the segmentation backbone, and StructureNet, which provides ground truth for hierarchies, bounding boxes, and part relationships. The study focused on three complex, highly structured categories: Chairs, Tables, and Storage Furniture.
Experimental Results
The interplay between these tasks yielded significant performance gains. In structure inference, the framework outperformed standard encoder-decoder baselines by a staggering 27.91% in part prediction accuracy (measured by Average Precision). In segmentation, the refinement step improved accuracy by 0.5% mAP. While the numerical gain in segmentation appears modest, qualitative results demonstrate a substantial reduction in "checkerboard" noise and illogical part splits.
Qualitative comparisons show that Seg&Struct produces much more diverse and shape-specific structures compared to baselines, which tend to collapse toward an "average" shape. Furthermore, the authors demonstrated that these predicted structures could be utilized for structure-aware shape retrieval, allowing for the identification of shapes that share a similar functional layout rather than just similar point cloud density.
Conclusion
Seg&Struct demonstrates that structure and geometry are mutually beneficial in 3D shape parsing. By forcing the model to reconcile local segments with global structural priors, the framework produces results that are both geometrically accurate and structurally sound.
A primary bottleneck identified by the authors is the presence of noisy annotations in existing datasets, where ambiguous boundaries (e.g., between a chair's "foot" and "leg") can confuse the refinement network. Future work may involve extending this framework to more diverse object categories or exploring unsupervised methods to learn structural priors, reducing the dependency on perfectly labeled hierarchical data.
Summary updated