Semantic and Context Information Fusion Network for View-Based 3D Model Classification and Retrieval

An-An Liu, Fu-Bin Guo, Heyu Zhou, Wenhui Li, Dan Song

Introduction

Identifying and retrieving 3D models—such as specific furniture or automotive parts from a database—is a fundamental challenge in computer vision. While 3D data is rich in information, it is computationally intensive. To address this, researchers often employ view-based methods, which project a 3D object into several 2D "snapshots" from various angles. These snapshots are then processed using established 2D image recognition techniques.

However, traditional view-based methods suffer from two primary limitations. First, they often use simple "pooling" (like max-pooling) to combine data from different views, which ignores the spatial context and the logical sequence of the views. Second, they frequently treat all pixels in a 2D snapshot as equally important, failing to distinguish between the object (semantic information) and irrelevant background noise.

The SCFN Architecture

The authors propose the Semantic and Context Information Fusion Network (SCFN) to overcome these hurdles. The SCFN treats a sequence of 2D views not as a disjointed collection of images, but as a cohesive narrative that describes a 3D shape.

The SCFN architecture, illustrating the flow from 2D view extraction to semantic and context information fusion.

Methods

The network is built around two novel modules designed to enhance feature extraction and fusion:

  • Channel Attention Mechanism (CAM): After extracting features from each 2D view using a standard backbone (such as AlexNet), the CAM acts as a "smart highlighter." It models correlations between different feature channels to assign higher weights to discriminative attributes (e.g., the specific curve of a car's wheel) while suppressing noise.
  • Context Information Fusion Module (CFM): Instead of basic pooling, the CFM utilizes a 3D Convolutional Neural Network (3D CNN). Since views are rendered in a specific sequence around the object, the 3D CNN captures the "flow" and appearance changes between adjacent views. This allows the model to learn the global spatial context of the 3D object more effectively than simple averaging.

Finally, a global average pooling layer produces a compact 3D descriptor used for classification via Softmax or retrieval via Euclidean distance.

Experimental Results

The SCFN was evaluated on three industry-standard benchmarks: ModelNet10, ModelNet40, and ShapeNetCore55.

Key Performance Metrics

  • Classification: On ModelNet10, the model achieved 94.1% accuracy, outperforming classic methods like MVCNN.
  • Retrieval: The gains were particularly significant in retrieval tasks. On the massive ShapeNetCore55 dataset (over 51,000 models), the SCFN saw improvements between 20.9% and 137.0% across various macro metrics (mAP and NDCG) compared to previous state-of-the-art algorithms.
  • Efficiency: High performance is achieved using only 12 views, making it more efficient than competing methods that require higher view counts or more complex multi-loop architectures.

Discussion and Insights

The core innovation of this work is the application of 3D CNNs to sequences of static 2D views. By treating these views like frames in a video, the network learns the spatial relationship between different perspectives (e.g., how a front-view transitions into a side-view). This sensitivity to the relationship between views allows the model to distinguish between very similar objects, such as a vase versus a bottle, which simple pooling methods often miss.

Limitations and Future Work

Currently, the SCFN relies entirely on 2D views and does not incorporate direct geometric data like point clouds or voxels. The authors suggest that future iterations could explore "multi-modal" fusion—combining optimized 2D view descriptors with raw 3D geometric information to create even more robust 3D representations.

Summary updated

Abstract

In recent years, with the rapid development of 3D technology, view-based methods have shown excellent performance in both 3D model classification and retrieval tasks. In view-based methods, how to aggregate multi-view features is a key issue. There are two commonly used solutions in the existing methods: 1) Use pooling strategy to merge multi-view features, but it ignores the context information contained in the continuous view sequence. 2) Leverage grouping strategy or long short term memory networks (LSTM) to select representative views of the 3D model, however, it easily neglects the semantic information of individual views. In this paper, we propose a novel Semantic and Context information Fusion Network (SCFN) to compensate for these drawbacks. First, we render views from multiple perspectives of the 3D model and extract the raw feature of the individual view by 2D convolutional neural networks (CNN). Then we design the channel attention mechanism (CAM) to exploit the view-wise semantic information. By modeling the correlation among view feature channels, we can assign higher weights to useful feature attributes, while suppressing the useless. Next, we propose a context information fusion module (CFM) to fuse multiple view features to obtain a compact 3D representation. Extensive experiments are conducted on three popular datasets, i.e., ModelNet10, ModelNet40, and ShapeNetCore55, which can demonstrate the superiority of the proposed method comparing to the state-of-the-arts on both 3D classification and retrieval tasks.
BibTeX
@article{liu2020semantic,
  author = {An-An Liu and Fu-Bin Guo and Heyu Zhou and Wenhui Li and Dan Song},
  title = {{Semantic and Context Information Fusion Network for View-Based 3D Model Classification and Retrieval}},
  journal = {IEEE Access},
  year = {2020},
  doi = {10.1109/access.2020.3018875},
}
Figure