3D Object Reconstruction from Imperfect Depth Data Using Extended YOLOv3 Network

Audrius Kulikajevas, Rytis Maskeliunas, Robertas Damasevicius, Edmond S. L. HoSensors 2020 · DOI · OpenReview

TL;DR

This paper presents a novel hybrid neural network that reconstructs 3D objects from imperfect depth data, even when parts are occluded. By extending YOLOv3 for object segmentation and integrating it with a specialized reconstruction network, they achieve improved accuracy and robustness, paving the way for more generalizable 3D scene understanding.

What Problem Does This Paper Address?

The paper tackles the challenge of 3D object reconstruction, a crucial technology for applications like smart manufacturing, virtual/augmented reality, and robotics. Existing methods often struggle with "imperfect depth data," meaning situations where depth sensors have limited coverage, leading to occluded objects and lost information. This is a common scenario in real-world applications, where obtaining complete 3D scans from all angles is often impractical or too expensive. The authors aim to reconstruct 3D objects accurately even with incomplete depth information, a significant step towards more practical and accessible 3D reconstruction systems.

Methods

The core of the proposed solution is a hybrid neural network architecture that combines object detection/segmentation with 3D reconstruction.

  • Extended YOLOv3 for Classification and Segmentation: They adapt the YOLOv3 object detection network to not only predict bounding boxes but also to output a geometric segmentation mask for each detected object. This allows for precise isolation of individual objects, even when they overlap.
  • Specialized Reconstruction Branches: For the 3D reconstruction part, they employ a modular approach. Instead of a single reconstruction network, they use multiple specialized branches, each trained for a specific object category. This allows for more accurate reconstruction tailored to the object's characteristics.
  • Depth-Aware Reconstruction Network: The reconstruction network itself is a deep convolutional neural network that takes the masked depth input. It utilizes a variational autoencoder (VAE) for latent space representation and transposed 3D convolutional layers for upsampling to generate a voxel cloud representation of the object.
  • Periodic Learning Rate: To address issues with training stability and falling into local minima, they introduce a periodic learning rate schedule that alternates between low and high rates, helping the model "jump out" of suboptimal solutions.
  • Dropout Layers: Additional dropout layers are incorporated to combat overfitting and improve generalization.

Datasets

  • ShapeNetCore: This dataset, a subset of ShapeNet, was used to generate synthetic 3D object models across 55 categories. These models were then used to create custom training datasets.
  • Real-world data: Intel RealSense ZR300 and D435i devices were used to acquire real-world depth data for visual validation.

The authors generated their own classification and reconstruction datasets by:

  • Classification Dataset: Creating random scenes with 1-10 objects from ShapeNetCore, ensuring uniform scaling and non-overlapping placement. Lighting was simulated using a Lambertian shading model.
  • Reconstruction Dataset: Rendering individual objects from multiple perspectives (48 views per object) to generate depth images and corresponding voxel cloud ground truths.

Key Insights and Ideas

  • Segmentation is Key for Imperfect Data: Precisely segmenting individual objects before reconstruction is crucial for handling occlusions and improving accuracy, especially when dealing with overlapping objects.
  • Modularity for Specialization: Using specialized reconstruction branches for different object categories allows for better adaptation and potentially higher accuracy compared to a single, general-purpose reconstruction network.
  • Depth Data Enhances Classification: While RGB data alone is often sufficient for object classification, combining it with depth information significantly improves recall rates and model generalization.
  • Periodic Learning Rate for Stability: The proposed periodic learning rate schedule is effective in improving model training stability and preventing it from getting stuck in local minima.
  • Real-world Applicability: The method demonstrates promising results on real-world data, indicating good generalization capabilities beyond synthetic datasets.

Key Results

  • Improved Reconstruction Accuracy: The proposed hybrid network achieved an 8.53% improvement in reconstruction results compared to the baseline YOLOv3 network.
  • More Precise Filling of Occluded Sides: The method allows for more precise filling of occluded object sides and reduction of noise during reconstruction.
  • Stable Reconstruction: The achieved reconstruction results are noted to be more stable compared to previous methods.
  • Object Instance Classification: The extended YOLOv3 achieved a mean Average Precision (mAP) of 60.20% for RGB-D input, outperforming the baseline YOLOv3 (55.75%).
  • Mask Prediction Similarity: The predicted masks showed high similarity to ground truth masks across different input types (Depth, RGB-D, RGB), with the proposed network achieving better similarity and lower standard deviation than the baseline YOLOv3.
  • Comparable to State-of-the-Art: The reconstruction results are comparable to other state-of-the-art methods like 3D-R2N2, reporting a mean IoU of 0.571.

Limitations and Future Work

  • Limited Scene Reconstruction: While the method excels at reconstructing individual objects, it currently lacks the ability to predict their relative positions, rotations, and scales within a scene for full scene reconstruction.
  • Perspective Invariance of Ground Truths: The ground truth data being perspective-invariant can slightly lower IoU metric values, even if the visual reconstruction is feasible.
  • Dataset Bias: The use of synthetic data from ShapeNetCore might introduce biases, and the authors suggest incorporating real-world datasets for further improvement.
  • Object Ambiguity: Some object classes in ShapeNetCore have significant overlap in form and function, making differentiation difficult, especially with low-resolution images.
  • Future Work Suggestions:
    • Investigate full scene reconstruction by finding camera space matrices.
  • Incorporate real-world depth sensor data into training.
  • Explore recurrent neural networks (RNNs) for processing sequential depth frames.
  • Combine RGB and depth frames more effectively to mitigate depth sensor noise.
  • Explore network architecture modifications like pruning dead neurons or using neuro-evolutionary algorithms.

Summary updated

Abstract

State-of-the-art intelligent versatile applications provoke the usage of full 3D, depth-based streams, especially in the scenarios of intelligent remote control and communications, where virtual and augmented reality will soon become outdated and are forecasted to be replaced by point cloud streams providing explorable 3D environments of communication and industrial data. One of the most novel approaches employed in modern object reconstruction methods is to use a priori knowledge of the objects that are being reconstructed. Our approach is different as we strive to reconstruct a 3D object within much more difficult scenarios of limited data availability. Data stream is often limited by insufficient depth camera coverage and, as a result, the objects are occluded and data is lost. Our proposed hybrid artificial neural network modifications have improved the reconstruction results by 8.53% which allows us for much more precise filling of occluded object sides and reduction of noise during the process. Furthermore, the addition of object segmentation masks and the individual object instance classification is a leap forward towards a general-purpose scene reconstruction as opposed to a single object reconstruction task due to the ability to mask out overlapping object instances and using only masked object area in the reconstruction process.
BibTeX
@article{kulikajevas2020d,
  author = {Audrius Kulikajevas and Rytis Maskeliunas and Robertas Damasevicius and Edmond S. L. Ho},
  title = {{3D Object Reconstruction from Imperfect Depth Data Using Extended YOLOv3 Network}},
  journal = {Sensors},
  year = {2020},
  doi = {10.3390/s20072025},
}
Figure