Abstract
In this paper, we propose a novel approach to solve the 3D non-rigid registration problem from RGB images using Convolutional Neural Networks (CNNs). Our objective is to find a deformation field that warps a given 3D canonical model into a novel instance observed by a single-view RGB image. This is done by training a CNN that infers a deformation field for the visible parts of the canonical model and by employing a learned shape (latent) space for inferring the deformations of the occluded parts. As a result of the registration, the observed model is reconstructed. Because our method does not need depth information, it can register objects that are typically hard to perceive with RGB-D sensors, such as those with transparent or shiny surfaces. Even without depth data, our approach outperforms the Coherent Point Drift (CPD) registration method for the evaluated object categories.
Introduction
A fundamental challenge in robotic perception is understanding the specific 3D geometry of novel objects to facilitate interaction. This is particularly difficult when objects are composed of materials like glass or reflective metal, which often baffle traditional depth sensors. This research, presented at IROS 2020, addresses the problem of "non-rigid registration"—finding a mathematical mapping (a deformation field) that transforms a generic template model into a specific observed instance.
Traditionally, this process requires high-quality 3D data from RGB-D sensors. However, these sensors fail on transparent or specular surfaces. By leveraging category-level prior knowledge, the proposed method performs 3D mapping using only standard 2D color images, effectively "filling in the blanks" for occluded parts of the object.
Methodology
The proposed pipeline consists of three main stages designed to infer 3D geometry from 2D cues:
- CNN-Based Deformation Inference: The system utilizes a modified FlowNet2 architecture. It processes four inputs: the observed RGB image, its bounding box mask, a rendered image of the canonical template in the same pose, and the template's mask. The network predicts a 3D deformation vector for every visible pixel.
- Latent Shape Space: To handle occluded regions, the authors pre-calculate a "latent space" of possible deformations for a category (e.g., spray bottles or drills) using Principal Component Analysis (PCA). This acts as a regularizer, ensuring the final model remains physically plausible within its category.
- Optimization and Zooming: The system optimizes the CNN’s predictions for visible parts against the latent shape space to reconstruct the full 3D model. To improve accuracy, a "zoom operation" crops and upsamples images based on bounding boxes, allowing the CNN to focus on fine geometric details.
Data Generation and Training
Due to the lack of standard datasets for intra-class 3D deformation, the authors developed a synthetic interpolation method. By morphing existing high-quality 3D meshes from databases like Sketchfab, they generated thousands of unique training instances with ground-truth deformation fields. This allowed the model to learn mappings for categories including spray bottles, cameras, and drills.
Key Results
The approach demonstrates significant improvements over traditional methods and robustness in challenging scenarios:
- Performance: The method significantly outperformed the standard Coherent Point Drift (CPD) algorithm. In camera test instances, the proposed method achieved an error of 102.17 µm, compared to 407.04 µm for CPD.
- Material Robustness: Because the system relies on visual appearance rather than light reflection or refraction, it successfully registers transparent glass bottles that typically appear as "holes" to standard 3D scanners.
- Pose Resilience: The system remains effective even when initial pose estimation is noisy, maintaining accuracy with errors up to ±0.05m/rad.
Conclusion
By combining CNN-driven local deformation inference with a global category-level shape space, this method enables accurate 3D reconstruction from single-view RGB images. This approach bypasses the limitations of depth sensors for transparent and reflective objects, providing a robust pathway for robots to interact with a wider variety of real-world items. Future work will focus on integrating pose refinement directly into the network and utilizing pixel-wise semantic segmentation for cluttered environments.
Summary updated