Multi-View Tree Structure Learning for 3D Model Retrieval and Classification in Smart City

An-An Liu, Zhenlan Zhao, Wenhui Li, Dan Song

Abstract

Efficient 3D model retrieval and classification are essential for managing the vast amounts of data generated in smart cities and digital manufacturing. Traditional view-based methods often suffer from information loss and order sensitivity. This paper introduces the Multi-View Tree Structure (MVTS) framework, which organizes 2D views into a content-aware tree structure using a Maximum Spanning Tree (MST) approach. By employing a bidirectional Tree-LSTM and an attention mechanism, MVTS captures complex spatial relationships and prioritizes informative views. Experimental results on ModelNet40 and ShapeNetCore55 demonstrate that MVTS achieves state-of-the-art performance, significantly outperforming existing sequence-based and pooling-based methods.

Introduction

In the era of smart cities and digital manufacturing, the volume of 3D data is expanding at an unprecedented rate. Efficiently searching for and categorizing these 3D models is critical for various applications, from urban planning to industrial design. Current 3D model processing techniques generally fall into two categories: model-based approaches, which process 3D voxels or point clouds directly, and view-based approaches, which utilize 2D images of the model. While model-based methods are computationally intensive, view-based methods offer efficiency but face three major hurdles: information loss from simple pooling, sensitivity to view order, and the varying importance of different viewing angles.

A complex 3D mechanical model representing the type of data encountered in digital manufacturing.

The Multi-View Tree Structure (MVTS) Framework

To address these challenges, the Multi-View Tree Structure (MVTS) learning method organizes 2D views into a content-aware tree rather than a simple sequence. This approach captures the complex spatial relationships between views without requiring fixed camera positions.

Methodology

The MVTS framework operates in three distinct stages:

  1. Visual Feature Learning: The system extracts features from 12 views of a 3D model using a pre-trained AlexNet backbone, providing a robust initial representation for each angle.
  2. Tree Model Construction: Rather than assuming a linear sequence, the method calculates a score matrix based on visual similarity between all pairs of views. Using Prim’s algorithm, a Maximum Spanning Tree (MST) is constructed to connect the most visually related views, effectively mapping the object's spatial flow. This MST is then converted into a Left-Child Right-Sibling (LCRS) tree for efficient computation.
  3. Tree Context Encoding:
    • Bidirectional Tree-LSTM: The model processes the tree structure in both top-down and bottom-up directions, allowing each view node to incorporate context from the entire model.
  4. Tree Attention: An attention module assigns weights to different views, prioritizing those that contribute most to the unique identity of the shape.
A 3D scanning device used to acquire multi-view data from physical objects.

Experimental Results

The MVTS method was rigorously evaluated on two primary benchmarks: ModelNet40 (12,311 CAD models) and ShapeNetCore55 (51,300 models using the official SHREC’17 split).

On the ModelNet40 dataset, MVTS achieved a 93.4% classification accuracy and an 87.3% retrieval mean Average Precision (mAP), outperforming established models such as MVCNN (90.1% accuracy) and GVCNN (90.7% accuracy). In the ShapeNetCore55 challenge, the method showed even more significant gains, particularly in macro metrics, where it outperformed existing state-of-the-art methods by margins ranging from 6.4% to 41.5%.

Key Insights and Discussion

  • Camera-Agnostic Learning: By building the tree structure based on visual content rather than a fixed camera path, the model remains robust regardless of camera placement.
  • Hierarchical Feature Representation: The Tree-LSTM architecture better reflects human perception of complex objects as a hierarchy of features rather than a flat sequence.
  • Robustness: Sensitivity analysis indicates that the choice of the root node in the tree construction does not significantly impact performance, validating the stability of the MST approach.

Future Directions

While MVTS sets new benchmarks, future work will explore the use of multiple trees with diverse visual features and more modern backbone architectures like transformers. Additionally, testing the framework on noisy, real-world 3D scans from smart city environments will be a key area for further empirical research.

Summary updated

Abstract

The application of digital products in smart city results in ever-increasing 3D model data and how to obtain the relevant 3D model becomes a crucial issue. In this paper, we propose the Multi-View Tree Structure (MVTS) learning for 3D model retrieval and recognition. MVTS contains three key consecutive modules. Firstly, the visual feature learning module extracts the visual features of multiple views. Then, we design a score matrix to estimate the value of contextual information between view pairs. Based on the score matrix, a maximum spanning tree is constructed to further explore the contextual information within multiple views. Then, we utilize the bidirectional Tree-LSTM to encode the contextual information among views and the spatial information of tree structure and optimize the tree parameters. After that, the tree attention strategy is adopted to explore the importance of each view. Comparing to existing methods, our proposed method explores the spatial information of 3D model without the requirement of specific camera settings, which is more suitable for real applications. Moreover, our method jointly realizes the feature learning, view-wise contextual information and tree spatial information encoding and view importance estimating, which enhances the discrimination of the 3D model representation. Extensive experimental results on Modelnet40 and ShapeNetCore55 demonstrate the superiority of our method.
BibTeX
@article{liu2020multi,
  author = {An-An Liu and Zhenlan Zhao and Wenhui Li and Dan Song},
  title = {{Multi-View Tree Structure Learning for 3D Model Retrieval and Classification in Smart City}},
  journal = {IEEE Access},
  year = {2020},
  doi = {10.1109/access.2020.3009333},
}
Figure