Deeply Exploiting Long-Term View Dependency for 3D Shape Recognition

Yong Xu, Chaoda Zheng, Ruotao Xu, Yuhui QuanIEEE Access 2019 · OpenReview

Abstract

Recognition of 3D shapes is a fundamental task in computer vision. In recent years, view-based deep learning has emerged as an effective approach. However, most existing methods treat the views of an object as an unordered set, ignoring dynamic relations like sequential semantic dependencies. This paper proposes modeling views as a sequence to exploit long-term dependencies using a sequence-aware view aggregation module based on a bi-directional Long Short-Term Memory (Bi-LSTM) network. This module captures bi-directional dependencies and is robust to circular shifts. Evaluated on ModelNet and ShapeNet datasets, the method demonstrates significant performance gains in classification and retrieval.

Introduction

3D object recognition is a cornerstone of modern computer vision, essential for applications like autonomous driving and robotics. View-based recognition, which represents 3D objects through a collection of 2D images, has proven highly effective. Despite its success, a major limitation persists: most frameworks (e.g., MVCNN) treat these views as an unordered set. By applying simple max or average pooling, these methods discard the spatial and sequential context inherent in how views are captured. This paper addresses this gap by treating views as a sequence, allowing the model to learn the "dynamic relations" between different angles of an object.

Methodology

The proposed framework utilizes a three-stage pipeline to process 3D shapes.

The proposed pipeline for 3D shape recognition, from multi-view representation to classification.
  1. Multi-View Representation: The system renders 12 views of a 3D model by placing virtual cameras at 30-degree intervals around the object's principal axis.
  2. Feature Extraction: Each view is processed by a shared VGG-11 Convolutional Neural Network (CNN) with Batch Normalization to extract high-level feature vectors.
  3. Sequence-Aware View Aggregation: This is the core innovation. Instead of pooling, the feature vectors are fed into a Bi-directional LSTM (Bi-LSTM). The Bi-LSTM processes the sequence in both forward and backward directions, enabling the network to link distant but semantically related views (e.g., the front and back of a car).

Key Insights: Symmetry and Sequentiality

The primary insight of this work is that geometric properties like symmetry manifest as long-range correlations in a view sequence. A Bi-LSTM is uniquely suited to "bridge" these dependencies.

Illustration of the circular sequence of views, highlighting the relationship between different camera angles.

Furthermore, the architecture is naturally robust to "circular shifts." Regardless of which angle the camera starts at, the Bi-LSTM captures the full global context. This makes the representation invariant to the initial camera orientation, a significant advantage over standard unidirectional LSTMs.

Experimental Results

The model was benchmarked against standard datasets:

  • ModelNet40/10: The method achieved 94.65% accuracy on ModelNet40 and 95.26% on ModelNet10, outperforming MVCNN (90.1%) and GVCNN (93.1%).
  • ShapeNetCore55: In large-scale retrieval tasks, the method achieved a micro-mAP of 0.778, showing superior performance in macro-metrics compared to previous state-of-the-art models like RotationNet.

Ablation studies confirmed that the Bi-LSTM is critical; replacing it with standard pooling or a unidirectional LSTM resulted in a performance drop of 1-2%.

Conclusion

By treating 3D views as a sequence rather than a set, this research demonstrates that long-term dependencies are vital for accurate shape recognition. The Bi-LSTM aggregation module provides a robust and efficient way to capture these relations. Future work may explore attention mechanisms to further refine view prioritization and handle more arbitrary camera viewpoints.

Summary updated

Abstract

Recognition of 3D shapes is a fundamental task in computer vision. In recent years, view-based deep learning has emerged as an effective approach for 3D shape recognition. Most existing view-based methods treat the views of an object as an unordered set, which ignores the dynamic relations among the views, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">e.g.</i> sequential semantic dependencies. In this paper, modeling the views of an object by a sequence, we aim at exploiting the long-term dependencies among different views for shape recognition, which is done by constructing a sequence-aware view aggregation module based on the bi-directional Long Short-Term Memory network. It is shown that our view aggregation module not only captures the bi-directional dependencies in view sequences, but also enjoys the robustness to circular shifts of input sequences. Incorporating the aggregation module into a standard convolutional network architecture, we develop an effective method for 3D shape classification and retrieval. Our method was evaluated on the ModelNet40/10 and ShapeNetCore55 datasets. The results show the encouraging performance gain from exploiting long-term dependencies in view sequences, as well as the superior performance of our method compared to the existing ones.
BibTeX
@inproceedings{xu2019deeply,
  author = {Yong Xu and Chaoda Zheng and Ruotao Xu and Yuhui Quan},
  title = {{Deeply Exploiting Long-Term View Dependency for 3D Shape Recognition}},
  booktitle = {IEEE Access},
  year = {2019},
  url = {https://openreview.net/forum?id=3HcwYOpa8S},
}
Figure