Fine-Grained 3D Shape Classification With Hierarchical Part-View Attention

Xinhai Liu, Zhizhong Han, Yu-Shen Liu, Matthias Zwicker

TL;DR

This paper introduces FG3D-Net, a novel deep learning architecture for fine-grained 3D shape classification. It addresses the challenge of distinguishing subtle differences between subcategories within broader object classes by leveraging hierarchical part-view attention and a new fine-grained 3D dataset (FG3D) to capture intricate local details from multiple object views.

What Problem Does This Paper Address?

The core problem this paper tackles is fine-grained 3D shape classification. While general 3D object recognition has seen significant progress, classifying objects that belong to the same broad category but have subtle differences (e.g., different types of airplanes like fighters vs. seaplanes) remains a significant challenge. Existing methods often struggle to capture these fine-grained details, especially when dealing with the small variances between subcategories. A key bottleneck identified is the lack of suitable benchmarks specifically designed for this fine-grained task.

Methods

FG3D-Net employs a multi-stage approach:

  • Sequential View Capturing: Multiple 2D renderings of a 3D object are generated from different viewpoints.
  • Region Proposal Network (RPN): An RPN is used to detect "generally semantic parts" (GSPs) within these rendered views. GSPs are local visual parts that don't necessarily correspond to specific semantic categories but highlight important visual cues. This is trained using existing 3D segmentation benchmarks.
  • Hierarchical Part-View Attention Aggregation: This is the core innovation. It comprises:
  • Part-Level Attention: Aggregates features from detected GSPs within a single view, highlighting the most discriminative parts.
  • View-Level Attention: Aggregates features from different views, emphasizing the most informative views for classification.
  • Recurrent Neural Network (RNN): A GRU-based RNN is used to capture the spatial relationships and temporal dependencies between sequential views, enhancing the global shape representation.
  • Fine-Grained Classification: The aggregated features are fed into a final classification layer to predict the subcategory.

Datasets

The authors introduce a new benchmark dataset called FG3D. This dataset consists of three main categories: Airplane, Car, and Chair, with a total of 66 subcategories. Unlike existing large-scale 3D datasets like ShapeNet, FG3D is specifically curated for fine-grained classification, ensuring each object is strictly assigned to a single subcategory. The dataset is constructed from various online repositories and organized using the WordNet taxonomy. For training and evaluation, the dataset is split into training and testing sets.

Key Insights and Ideas

  • The Importance of "Generally Semantic Parts" (GSPs): The paper argues that focusing on these local visual cues, rather than predefined semantic parts, is crucial for capturing fine-grained details that differentiate subcategories.
  • Hierarchical Attention is Key: The combination of part-level and view-level attention is essential. Part-level attention refines feature extraction within a view, while view-level attention selects the most discriminative views, leading to a more robust global representation.
  • Sequential View Information Matters: Incorporating an RNN to model the spatial relationships between views significantly improves performance, highlighting the value of temporal context in multi-view 3D analysis.
  • A Dedicated Fine-Grained Dataset is Necessary: The introduction of FG3D demonstrates the limitations of existing datasets for this specific task and provides a valuable resource for future research.

Key Results

FG3D-Net achieves state-of-the-art performance on the proposed FG3D dataset. For instance, on the Airplane category, it reaches an accuracy of 89.44% for instance classification and 80.04% for class classification. These results significantly outperform other methods, including point-based, voxel-based, and other view-based approaches. The paper also shows that using 12 views is sufficient for achieving high accuracy, and that the GRU cell type for the RNN is superior to other options.

Limitations and Future Work

The authors acknowledge several limitations:

  • Limited Number of GSPs: The current method relies on GSPs detected from existing segmentation benchmarks, which may not cover all possible fine-grained parts.
  • Small Dataset Size: The FG3D dataset, while new, has a relatively small number of categories (three) and subcategories compared to broader 3D datasets. This limits the generalizability of the model to unseen categories.
  • No Validation Set: The dataset split lacks a dedicated validation set, which could impact hyperparameter tuning.

Future work suggested includes:

  • Exploring unsupervised methods for GSP detection to overcome the reliance on external segmentation data.
  • Expanding the FG3D dataset with more categories and subcategories.
  • Investigating ways to improve performance on a larger number of shape categories.

Summary updated

Abstract

Fine-grained 3D shape classification is important for shape understanding and analysis, which poses a challenging research problem. However, the studies on the fine-grained 3D shape classification have rarely been explored, due to the lack of fine-grained 3D shape benchmarks. To address this issue, we first introduce a new 3D shape dataset (named FG3D dataset) with fine-grained class labels, which consists of three categories including airplane, car and chair. Each category consists of several subcategories at a fine-grained level. According to our experiments under this fine-grained dataset, we find that state-of-the-art methods are significantly limited by the small variance among subcategories in the same category. To resolve this problem, we further propose a novel fine-grained 3D shape classification method named FG3D-Net to capture the fine-grained local details of 3D shapes from multiple rendered views. Specifically, we first train a Region Proposal Network (RPN) to detect the generally semantic parts inside multiple views under the benchmark of generally semantic part detection. Then, we design a hierarchical part-view attention aggregation module to learn a global shape representation by aggregating generally semantic part features, which preserves the local details of 3D shapes. The part-view attention module hierarchically leverages part-level and view-level attention to increase the discriminability of our features. The part-level attention highlights the important parts in each view while the view-level attention highlights the discriminative views among all the views of the same object. In addition, we integrate a Recurrent Neural Network (RNN) to capture the spatial relationships among sequential views from different viewpoints. Our results under the fine-grained 3D shape dataset show that our method outperforms other state-of-the-art methods. The FG3D dataset is available at https://github.com/liuxinhai/FG3D-Net.
BibTeX
@article{liu2021fine,
  author = {Xinhai Liu and Zhizhong Han and Yu-Shen Liu and Matthias Zwicker},
  title = {{Fine-Grained 3D Shape Classification With Hierarchical Part-View Attention}},
  journal = {IEEE Trans. Image Process.},
  year = {2021},
  url = {https://openreview.net/forum?id=fNWYEKUSHQY},
}
Figure