Joint Multi-view 2D Convolutional Neural Networks for 3D Object Classification

Jinglin Xu, Xiangsen Zhang, Wenbin Li, Xinwang Liu, Junwei Han

Introduction

3D object classification is a fundamental task in computer vision, with critical applications in autonomous driving and robotics. While direct processing of 3D data (e.g., voxels or point clouds) is intuitive, it is often computationally expensive and hindered by data sparsity. View-based methods, which project 3D objects into multiple 2D images, offer a more efficient alternative. However, traditional approaches like Multi-View CNN (MVCNN) often treat these views as independent or rely on simple pooling, neglecting the intrinsic spatial and content relationships between different camera angles.

This paper proposes a novel framework that explicitly models "inter-view" relationships and employs an adaptive weighting strategy to prioritize informative views while discarding redundant ones.

Methodology

The proposed architecture integrates three primary components to capture both local and global object features:

  1. Intra-view Feature Extraction: The system renders 12 2D views of a 3D object at 30-degree intervals. Each view is processed through a shared ResNet-18 backbone to extract high-level individual feature descriptors.
  2. Inter-view Information Calculation: To capture structural correlations, the model calculates the outer product between features of different view pairs. These correlation matrices are then processed via view pair pooling and 1D convolutions to generate a comprehensive inter-view descriptor for each perspective.
  3. Selective and Adaptive Loss Fusion: Recognizing that not all views are equally discriminative, the authors formulate a constrained optimization problem using an $\ell_0$ norm constraint. This mechanism adaptively assigns weights to views, effectively "zeroing out" uninformative or noisy perspectives during the final classification.
Overview of the proposed joint multi-view 2D-CNN architecture, illustrating the flow from view rendering to selective loss fusion.

Experimental Setup

The model was evaluated on the ModelNet40 benchmark, which contains 12,311 CAD models across 40 categories. For each model, 12 virtual camera views were rendered at 30-degree increments around the vertical axis. The primary metric for evaluation is classification accuracy.

Results and Discussion

The proposed method, denoted as Ours (ResNet-18), achieves a state-of-the-art accuracy of 94.16% on ModelNet40. As shown in the comparison below, this approach outperforms several high-profile baselines, including standard MVCNN and GVCNN implementations.

Comparison of classification accuracy between the proposed method and existing multi-view baselines on ModelNet40.

Ablation studies indicate that the inclusion of inter-view information and the selective weighting strategy are both critical to the performance gains. Specifically, the full model outperforms a baseline using only feature concatenation (87.92%) by a significant margin, demonstrating the value of modeling view correlations. Visualization of learned weights further confirms that the model successfully identifies and prioritizes the most descriptive views of an object, such as the side profiles of a chair versus its bottom.

Conclusion

By treating 3D classification as a joint 2D image integration task, this research demonstrates that modeling inter-view correlations and implementing selective view fusion can significantly enhance performance. The approach leverages the maturity of 2D-CNN architectures while introducing a sophisticated logic for multi-view synthesis, achieving 94.16% accuracy on the ModelNet40 benchmark. Future work may explore extending this framework to arbitrary viewpoints and more complex tasks like 3D object retrieval.

Summary updated

Abstract

Three-dimensional (3D) object classification is widely involved in various computer vision applications, e.g., autonomous driving, simultaneous localization and mapping, which has attracted lots of attention in the committee. However, solving 3D object classification by directly employing the 3D convolutional neural networks (CNNs) generally suffers from high computational cost. Besides, existing view-based methods cannot better explore the content relationships between views. To this end, this work proposes a novel multi-view framework by jointly using multiple 2D-CNNs to capture discriminative information with relationships as well as a new multi-view loss fusion strategy, in an end-to-end manner. Specifically, we utilize multiple 2D views of a 3D object as input and integrate the intra-view and inter-view information of each view through the view-specific 2D-CNN and a series of modules (outer product, view pair pooling, 1D convolution, and fully connected transformation). Furthermore, we design a novel view ensemble mechanism that selects several discriminative and informative views to jointly infer the category of a 3D object. Extensive experiments demonstrate that the proposed method is able to outperform current state-of-the-art methods on 3D object classification. More importantly, this work provides a new way to improve 3D object classification from the perspective of fully utilizing well-established 2D-CNNs.
BibTeX
@inproceedings{xu2020joint,
  author = {Jinglin Xu and Xiangsen Zhang and Wenbin Li and Xinwang Liu and Junwei Han},
  title = {{Joint Multi-view 2D Convolutional Neural Networks for 3D Object Classification}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.24963/ijcai.2020/443},
}
Figure