MV-C3D: A Spatial Correlated Multi-View 3D Convolutional Neural Networks

Qi Xuan, Fuxian Li, Yi Liu, Yun Xiang

Introduction

Recognizing 3D objects is a fundamental challenge in computer vision. Traditional approaches typically fall into two categories: voxel-based methods, which represent objects as 3D grids, and multi-view methods, which utilize 2D snapshots from various angles. While multi-view methods are generally more efficient, they often suffer from two significant limitations. First, they frequently assume access to a full 360-degree set of views, which is impractical for real-world applications like robotics or fixed-camera surveillance. Second, they often ignore the intrinsic spatial correlation between adjacent views, processing them independently before a final pooling stage.

MV-C3D addresses these gaps by treating a sequence of 2D images as a spatially correlated stack, allowing the model to understand the object's geometry through contiguous viewpoints.

Overview of the MV-C3D framework, illustrating the transition from 36 viewpoints to a contiguous image stack processed via 3D convolutions.

Methodology

The primary innovation of this work is the Multi-View 3D Convolutional Neural Network (MV-C3D) architecture. Instead of processing images individually through a 2D network, MV-C3D stacks $N$ contiguous images into a "feature cube."

3D Convolutional Layers

By employing 3D kernels (typically $3 \times 3 \times 3$), the network simultaneously learns features across the spatial dimensions (height and width) and the viewpoint dimension. This enables the model to capture how specific object features evolve as the camera's perspective shifts.

Mechanism of 3D convolution applied to a sequence of views to extract spatio-temporal features.

Architecture and Training

The model architecture is inspired by VGGNet, consisting of eight 3D convolutional layers, five 3D max-pooling layers, and three fully connected layers. To overcome the lack of massive multi-view 3D datasets, the authors utilized a transfer learning strategy, pre-training the model on the UCF101 video action recognition dataset. This is effective because video frames, like multi-view stacks, exhibit strong spatial and temporal correlations. Additionally, oversampling was used to address class imbalances within the training data.

Experimental Evaluation

The performance of MV-C3D was benchmarked against three major datasets:

  1. ModelNet10 & ModelNet40: Standard CAD model benchmarks for 3D object recognition.
  2. MIRO (Multi-view Images of Rotated Objects): A real-world dataset featuring 120 object instances captured from various angles, providing a test for robustness against real-world noise.

Key Results

MV-C3D achieved state-of-the-art results, notably reaching 93.9% accuracy on ModelNet40, surpassing the standard MVCNN by 3.8%. On the real-world MIRO dataset, it achieved 93.3% accuracy, significantly outperforming MVCNN (83.3%) and DSCNN (87.5%).

A critical finding was the model's robustness to partial data. MV-C3D maintains high accuracy (over 89%) with as few as 10 contiguous views. This demonstrates that the network learns the underlying 3D structure rather than merely memorizing 2D patterns.

Conclusion and Future Work

MV-C3D demonstrates that treating viewpoints as a correlated dimension significantly enhances 3D object recognition, especially when only partial, contiguous data is available. While the model achieves high accuracy, its complexity—approximately 299 million parameters—poses challenges for deployment on edge devices. Future research will focus on optimizing the architecture to reduce parameter counts and refining convolution patterns for even sparser viewpoint ranges.

Summary updated

Abstract

As the development of deep neural networks, 3D object recognition is becoming increasingly popular in computer vision community. Many multi-view based methods are proposed to improve the category recognition accuracy. These approaches mainly rely on multi-view images which are rendered with the whole circumference. In real-world applications, however, 3D objects are mostly observed from partial viewpoints in a less range. Therefore, we propose a multi-view based 3D convolutional neural network, which takes only part of contiguous multi-view images as input and can still maintain high accuracy. Moreover, our model takes these view images as a joint variable to better learn spatially correlated features using 3D convolution and 3D max-pooling layers. Experimental results on ModelNet10 and ModelNet40 datasets show that our MV-C3D technique can achieve outstanding performance with multi-view images which are captured from partial angles with less range. The results on 3D rotated real image dataset MIRO further demonstrate that MV-C3D is more adaptable in real-world scenarios. The classification accuracy can be further improved with the increasing number of view images.
BibTeX
@article{xuan2019mv,
  author = {Qi Xuan and Fuxian Li and Yi Liu and Yun Xiang},
  title = {{MV-C3D: A Spatial Correlated Multi-View 3D Convolutional Neural Networks}},
  journal = {IEEE Access},
  year = {2019},
  doi = {10.1109/access.2019.2923022},
}
Figure