A Novel Patch Convolutional Neural Network for View-based 3D Model Retrieval

Zan Gao, Yuxiang Shao, Weili Guan, Meng Liu, Zhiyong Cheng, Shengyong Chen

Abstract

View-based 3D model retrieval often fails to capture the intricate relationships between local image patches across different perspectives. This paper introduces the Patch Convolutional Neural Network (PCNN), which treats patches from multiple views as nodes in a feature-space graph. By utilizing a PatchConv layer for relational learning and an Adaptive Weighted View (AWV) layer for intelligent view fusion, PCNN achieves state-of-the-art performance, notably reaching a 93.67% mAP on the ModelNet40 benchmark.

Introduction

3D model retrieval—the process of identifying a specific 3D object within a database using a query—is a fundamental task in computer vision and industrial design. Among various approaches, view-based methods have emerged as highly effective. These methods typically render multiple 2D snapshots of a 3D object from various angles and aggregate them into a single descriptor for classification or retrieval.

However, traditional view-based methods possess a significant "blind spot": they often treat each 2D view as an independent unit or rely on simple pooling mechanisms (such as max-pooling) to combine them. This approach overlooks the latent relationships between local parts (patches) across different views. For instance, a patch depicting an airplane wing in a side profile is intrinsically linked to a patch of the same wing in a top-down view. Standard models fail to capture these long-range, cross-view associations, resulting in a loss of discriminative detail.

The PCNN Framework

To address these limitations, the authors propose the Patch Convolutional Neural Network (PCNN). The framework integrates three innovative modules into a unified pipeline designed to mine local patch relationships.

The overall architecture of the Patch Convolutional Neural Network (PCNN), highlighting the PatchConv and Adaptive Weighted View layers.

1. Patch Feature Extraction

The process begins with a ResNet34 backbone that extracts $7 \times 7$ feature patches from each of the 12 rendered views of a 3D object. This shifts the focus from global image features to localized descriptors.

2. PatchConv Layer

The PatchConv layer is the core innovation of the PCNN. Inspired by graph neural networks, it treats all patches from all views as nodes in a single graph. Using a k-nearest neighbor (k-NN) algorithm in the feature space, the layer identifies related patches regardless of their original view. It then applies a convolution-like operation to extract relational information between these nodes. This allows the model to "connect the dots" between similar parts of an object seen from disparate perspectives, effectively compensating for information lost in any single, poorly-angled view.

3. Adaptive Weighted View (AWV) Layer

Recognizing that not all views are equally informative, the AWV layer employs a self-attention mechanism. It compares each view's features against a global "view-pooling" feature to automatically assign higher weights to the most representative views. This ensures that the final descriptor is dominated by the most discriminative visual information.

4. Discrimination Loss Function

Training is guided by a composite loss function that includes a "Weighted View Loss." This mechanism performs "hard-view mining" by forcing the network to pay more attention to views that are significantly different from the average, which are typically the most difficult to classify.

Experimental Results

The PCNN was evaluated on two primary benchmarks: ModelNet40 (12,311 models, 40 categories) and ModelNet10 (4,899 models, 10 categories).

Precision-Recall curves on the ModelNet40 dataset comparing PCNN against various baselines and ablation versions.

The model achieved a Mean Average Precision (mAP) of 93.67% on ModelNet40 and 96.23% on ModelNet10, outperforming established architectures such as MVCNN and GVCNN. Ablation studies confirmed the necessity of the proposed modules; removing either the PatchConv or AWV layers resulted in a measurable drop in mAP, proving that capturing cross-view patch relationships is the "secret sauce" behind the model's success.

Conclusion

The PCNN framework demonstrates that shifting the focus from entire views to local patch relationships significantly enhances 3D model retrieval. By mining feature-space connectivity and employing intelligent view weighting, the model achieves state-of-the-art results. Future work may focus on optimizing the computational complexity of the k-NN graph construction and exploring performance with arbitrary, non-fixed camera viewpoints.

Summary updated

Abstract

In industrial enterprises, effective retrieval of three-dimensional (3-D) computer-aided design (CAD) models can greatly save time and cost in new product development and manufacturing, thus, many researchers have focused on it. Recently, many view-based 3D model retrieval methods have been proposed and have achieved state-of-the-art performance. However, most of these methods focus on extracting more discriminative view-level features and effectively aggregating the multi-view images of a 3D model, and the latent relationship among these multi-view images is not fully explored. Thus, we tackle this problem from the perspective of exploiting the relationships between patch features to capture long-range associations among multi-view images. To capture associations among views, in this work, we propose a novel patch convolutional neural network (PCNN ) for view-based 3D model retrieval. Specifically, we first employ a CNN to extract patch features of each view image separately. Second, a novel neural network module named PatchConv is designed to exploit intrinsic relationships between neighboring patches in the feature space to capture long-range associations among multi-view images. Then, an adaptive weighted view layer is further embedded into PCNN to automatically assign a weight to each view according to the similarity between each view feature and the view-pooling feature. Finally, a discrimination loss function is employed to extract the discriminative 3D model feature, which consists of softmax loss values generated by the fusion classifier and the specific classifier. Extensive experimental results on two public 3D model retrieval benchmarks, namely, the ModelNet40, and ModelNet10, demonstrate that our proposed PCNN can outperform state-of-the-art approaches, with mAP values of 93.67%, and 96.23%, respectively.
BibTeX
@inproceedings{gao2021novel,
  author = {Zan Gao and Yuxiang Shao and Weili Guan and Meng Liu and Zhiyong Cheng and Shengyong Chen},
  title = {{A Novel Patch Convolutional Neural Network for View-based 3D Model Retrieval}},
  booktitle = {ACM Multimedia},
  year = {2021},
  url = {https://openreview.net/forum?id=AwKZEMb1-Ar},
}
Figure