Abstract
We present a method that propagates point-wise feature representations across shapes within a collection for the purpose of 3D shape segmentation. This is achieved through a cross-shape attention operation that assesses the degree of interaction between points on different shapes and mediates feature propagation. For each test shape, our method finds shapes in an input collection that are suited for executing such cross-shape attention operations. The resulting point-wise feature representations lead to more consistent 3D shape segmentation results, as demonstrated in our experiments.
Introduction
3D part segmentation—the task of assigning semantic labels to every point in a 3D model—is a fundamental challenge in computer vision. Traditional deep learning approaches typically process 3D shapes in isolation, learning features based on local spatial relationships within a single object. However, individual shapes are often plagued by noise, incompleteness, or geometric ambiguity. This research introduces a "collection-aware" paradigm, where neural networks leverage prior knowledge from similar objects within a dataset to resolve local uncertainties and produce more consistent segmentations.
Methodology: Cross-Shape Attention
The core innovation is the Cross-Shape Attention (CSA) layer, a modular component designed to be integrated into existing 3D backbones such as MinkowskiNet or MID-FC. The framework operates through three primary stages:
- Key Shape Retrieval: For any given "query" shape, the system identifies a subset of $K$ "key" shapes from the collection that exhibit high geometric similarity. This is achieved using a specialized retrieval metric that evaluates the average similarity between the best-matching points of two shapes.
- Cross-Shape Attention Layer: Drawing inspiration from Transformer architectures, this layer computes Query, Key, and Value representations for points across shapes. It establishes a point-to-point attention matrix, allowing a point on the query shape to "borrow" or synchronize features from corresponding points on the retrieved key shapes.
- Shape Compatibility: To ensure robust feature propagation, the model learns a compatibility function that weights the influence of different key shapes. This ensures that highly similar objects have a proportionally stronger impact on the final feature representation.
Experimental Evaluation
The proposed method was rigorously evaluated on the PartNet dataset, a large-scale benchmark featuring fine-grained part annotations for thousands of 3D objects across 24 categories.
Quantitative and Qualitative Results
The integration of the CSA layer led to significant performance gains across multiple categories:
- State-of-the-Art Performance: The method established a new record on the PartNet benchmark with a mean Part IoU (Intersection over Union) of 62.1%.
- Architecture Versatility: The CSA module provided a 3.1% improvement over the MinkowskiNet baseline and a 1.3% boost to the MID-FC architecture, demonstrating its effectiveness as a general-purpose enhancement for 3D segmentation.
- Visual Consistency: Qualitative analysis reveals that the model produces significantly cleaner part boundaries. By referencing well-defined examples in the training set, the network can accurately identify thin structures or sparse geometric features that isolated models often misclassify.
Discussion and Future Directions
While the CSA framework offers substantial improvements in accuracy and consistency, it introduces additional computational complexity. The cross-shape operation scales quadratically with the number of points, and constructing the initial shape graph is an $O(N^2)$ operation relative to the collection size. Future research will focus on optimizing these operations through sparsified attention mechanisms and exploring the application of cross-shape synchronization to larger-scale 3D scenes.
Summary updated