Learning Generalizable Part-based Feature Representation for 3D Point Clouds

Xin Wei, Xiang Gu, Jian SunNeurIPS 2022 · OpenReview

The Challenge of 3D Domain Shift

Deep learning models for 3D point clouds have achieved remarkable success in classification tasks. However, these models are notoriously brittle when deployed in the real world. A classifier trained on clean, synthetic CAD models (like those in ModelNet) often fails when tested on real-world data (like ScanObjectNN) captured by LiDAR or depth sensors.

This failure stems from domain shift—differences in point density, sensor noise, and self-occlusion between training and testing environments. While domain generalization in 2D images is well-studied, 3D point clouds present unique geometric challenges.

Conceptual overview of mapping source and target parts to a shared template space.

The authors of this paper observed a crucial geometric property: while a whole object's shape might change drastically between domains due to occlusions or noise, its local components (like the legs of a chair or the handle of a mug) remain much more consistent. The challenge is how to represent these parts so that a model can generalize across unseen environments. To solve this, they introduce the Part-based Domain Generalization (PDG) network.

The PDG Architecture

The PDG network shifts the focus from global shapes to local geometric parts, utilizing a shared "part-template" space to bridge the domain gap. The architecture consists of several novel stages:

The architecture of the Part-based Domain Generalization (PDG) network.
  1. Part Decomposition: Instead of processing the entire point cloud at once, the model uses Farthest Point Sampling (FPS) and K-Nearest Neighbors (KNN) to break the shape into overlapping local parts.
  2. Part-Template Feature Space: The core innovation is a set of learnable "templates"—essentially a shared visual vocabulary of geometric primitives.
  3. Feature Alignment via Cross-Attention: The model employs a cross-attention mechanism to map raw local features to these templates. This aligns features from different domains into a common, generalizable representation, effectively ignoring domain-specific noise.
  4. Confidence-based Aggregation: Recognizing that not all parts are equally informative (e.g., a flat surface vs. a distinct wheel), the model learns to score each part's "discriminative power." It weights them accordingly before pooling them into a final global shape representation.
  5. Multi-scale Contrastive Learning: To further improve robustness, the authors implement a contrastive loss at both the part and shape levels. This forces the model to produce similar features for the same object even when subjected to corruptions like point dropping or noise.

Key Insights and Results

The foundational insight of PDG is that local geometry is more stable than global geometry. The authors proved this empirically using the $\mathcal{A}$-distance (a measure of distribution discrepancy), demonstrating that the domain gap for small parts is significantly smaller than for the global shape.

By forcing the network to describe parts as combinations of learned templates, the model anchors its understanding in structural signals rather than domain-specific artifacts. Furthermore, using cross-attention to align features to templates proved more effective and computationally efficient than traditional alignment methods like Optimal Transport.

When evaluated on major 3D domain generalization benchmarks, PDG sets a new state-of-the-art:

  • Massive Gains on Real Data: On the challenging Sim-to-Real task (training on synthetic ModelNet and testing on real-world ScanObjectNN), PDG achieved 67.6% accuracy, outperforming the previous state-of-the-art (MetaSets) by over 7%.
  • Consistent Superiority: Across various synthetic-to-real tasks in the 3DDG benchmark, PDG consistently outperformed standard backbones and specialized domain adaptation methods.
  • Ablation Success: Removing the contrastive learning or the part-based aggregation led to noticeable performance drops, confirming that the synergy of these components drives the model's robustness.

Limitations and Future Work

While highly effective, the current PDG framework focuses strictly on classification. The authors plan to extend this part-based logic to more complex tasks, such as 3D object detection and semantic segmentation. Additionally, breaking a point cloud into many parts and running cross-attention adds computational complexity compared to simple global pooling. Future work may also explore how these part-templates scale when dealing with much larger, more diverse datasets beyond standard category limits.

Summary updated

Abstract

Deep networks on 3D point clouds have achieved remarkable success in 3D classification, while they are vulnerable to geometry variations caused by inconsistent data acquisition procedures. This results in a challenging 3D domain generalization (3DDG) problem, that is to generalize a model trained on source domain to an unseen target domain. Based on the observation that local geometric structures are more generalizable than the whole shape, we propose to reduce the geometry shift by a generalizable part-based feature representation and design a novel part-based domain generalization network (PDG) for 3D point cloud classification. Specifically, we build a part-template feature space shared by source and target domains. Shapes from distinct domains are first organized to part-level features and then represented by part-template features. The transformed part-level features, dubbed aligned part-based representations, are then aggregated by a part-based feature aggregation module. To improve the robustness of the part-based representations, we further propose a contrastive learning framework upon part-based shape representation. Experiments and ablation studies on 3DDA and 3DDG benchmarks justify the efficacy of the proposed approach for domain generalization, compared with the previous state-of-the-art methods. Our code will be available on http://github.com/weixmath/PDG.
BibTeX
@inproceedings{wei2022learning,
  author = {Xin Wei and Xiang Gu and Jian Sun},
  title = {{Learning Generalizable Part-based Feature Representation for 3D Point Clouds}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=XgexGrV4FmM},
}
Figure