Fine-Grained 3D Shape Classification with Hierarchical Part-View Attentions

Xinhai Liu, Zhizhong Han, Yu-Shen Liu, Matthias Zwicker

Abstract

Fine-grained 3D shape classification is a critical yet challenging task in shape understanding, primarily due to the subtle geometric differences between subcategories. This paper addresses two major bottlenecks: the lack of high-quality fine-grained benchmarks and the inability of global feature extractors to capture local nuances. We introduce the FG3D dataset, a curated benchmark with 66 subcategories, and FG3D-Net, a novel framework that utilizes a Region Proposal Network (RPN) and hierarchical part-view attention to distinguish highly similar 3D models. Our approach achieves state-of-the-art performance by effectively aggregating local semantic details and capturing spatial relationships across multiple views.

Introduction

While 3D shape classification has advanced significantly with datasets like ModelNet and ShapeNet, most research focuses on "coarse-grained" tasks—distinguishing broad categories like chairs from cars. However, "fine-grained" classification—telling a Boeing 747 apart from a Tupolev Tu-154—is much harder because global silhouettes are often nearly identical. Standard methods tend to extract global features that "wash out" the tiny, local geometric differences (such as wingtip shapes or engine placements) that define a subcategory. To bridge this gap, we propose a method that focuses on "Generally Semantic Parts" (GSPs) and leverages hierarchical attention to prioritize the most discriminative features.

The FG3D Dataset

To facilitate research in this area, we introduced the FG3D dataset, a large-scale benchmark for fine-grained 3D shape classification. Unlike existing datasets that may have noisy or inconsistent labels, FG3D is a curated subset of ShapeNet and other repositories where each model is strictly assigned to one unique, verified subcategory label.

The FG3D dataset features fine-grained subcategories for airplanes, cars, and chairs, ensuring high label consistency for benchmarking.

The dataset consists of three primary categories—Airplane, Car, and Chair—encompassing 66 subcategories and approximately 25,000 3D models. This clean benchmark allows for a rigorous evaluation of a model's ability to handle small inter-class variance.

FG3D-Net Methodology

Our proposed architecture, FG3D-Net, processes 3D objects through a multi-view pipeline designed to preserve and highlight local details.

The FG3D-Net architecture: from sequential view capturing to hierarchical part-view attention aggregation and final classification.

1. Generally Semantic Part (GSP) Detection

Instead of analyzing a whole rendered view, FG3D-Net uses a Region Proposal Network (RPN) to identify semantic parts such as wheels, wings, or armrests. By focusing on these specific regions, the model can ignore common features shared by all members of a category and focus on the parts that actually vary between subcategories.

2. Hierarchical Part-View Attention

This core module applies attention at two distinct levels:

  • Part-level Attention: Within a single view, the model learns which detected parts (e.g., the specific curve of a wing) are most important for classification.
  • View-level Attention: Across all rendered views, it identifies which camera angles provide the most discriminative information, effectively filtering out redundant or uninformative perspectives.

3. Spatial Correlation with RNNs

Because the views are rendered in a sequence around the object, we integrate a Gated Recurrent Unit (GRU) to capture the spatial relationships between consecutive views. This ensures the model understands the object's continuous 3D structure rather than treating the views as an unordered collection of 2D images.

Experimental Results

FG3D-Net was evaluated against a wide array of competitors, including point-based (PointNet++), voxel-based (VoxNet), and view-based (MVCNN) models. Our method achieved superior instance accuracy across all three categories:

  • Airplanes: 93.99%
  • Chairs: 83.94%
  • Cars: 79.47%

Ablation studies confirmed that both the hierarchical attention mechanism and the RNN are vital; removing either component led to a significant degradation in performance. This proves that both local detail and spatial context are necessary for successful fine-grained recognition.

Conclusion

FG3D-Net demonstrates that focusing on semantic parts and their spatial relationships is key to mastering fine-grained 3D shape classification. While the model currently relies on pre-trained part detectors, future work could explore unsupervised part discovery to expand the system's applicability to more diverse object categories. The FG3D dataset remains a valuable resource for the community to push the boundaries of detailed 3D shape understanding.

Summary updated

Abstract

Fine-grained 3D shape classification is important for shape understanding and analysis, which poses a challenging research problem. However, the studies on the fine-grained 3D shape classification have rarely been explored, due to the lack of fine-grained 3D shape benchmarks. To address this issue, we first introduce a new 3D shape dataset (named FG3D dataset) with fine-grained class labels, which consists of three categories including airplane, car and chair. Each category consists of several subcategories at a fine-grained level. According to our experiments under this fine-grained dataset, we find that state-of-the-art methods are significantly limited by the small variance among subcategories in the same category. To resolve this problem, we further propose a novel fine-grained 3D shape classification method named FG3D-Net to capture the fine-grained local details of 3D shapes from multiple rendered views. Specifically, we first train a Region Proposal Network (RPN) to detect the generally semantic parts inside multiple views under the benchmark of generally semantic part detection. Then, we design a hierarchical part-view attention aggregation module to learn a global shape representation by aggregating generally semantic part features, which preserves the local details of 3D shapes. The part-view attention module hierarchically leverages part-level and view-level attention to increase the discriminability of our features. The part-level attention highlights the important parts in each view while the view-level attention highlights the discriminative views among all the views of the same object. In addition, we integrate a Recurrent Neural Network (RNN) to capture the spatial relationships among sequential views from different viewpoints. Our results under the fine-grained 3D shape dataset show that our method outperforms other state-of-the-art methods.
BibTeX
@article{liu2021fine,
  author = {Xinhai Liu and Zhizhong Han and Yu-Shen Liu and Matthias Zwicker},
  title = {{Fine-Grained 3D Shape Classification with Hierarchical Part-View Attentions}},
  journal = {IEEE Trans. Image Process.},
  year = {2021},
  url = {https://openreview.net/forum?id=fNWYEKUSHQY},
}
Figure