Category-Level 3D Non-Rigid Registration from Single-View RGB Images

Diego Rodriguez, Florian Huber, Sven Behnke

This paper, presented at IROS 2020, tackles a fundamental challenge in robotic perception: how can a robot understand the specific 3D geometry of a novel object well enough to interact with it, especially when that object is made of "difficult" materials like glass or shiny metal?

TL;DR

The authors present a deep learning approach for category-level 3D non-rigid registration using only a single RGB image. By combining a Convolutional Neural Network (CNN) with a learned "shape space," the system can warp a generic 3D model to fit a specific observed instance, effectively reconstructing occluded parts and handling transparent or reflective objects that baffle traditional depth sensors.

What Problem Does This Paper Address?

In robotics, "non-rigid registration" is the process of finding a mathematical mapping (a deformation field) that transforms a template model into a specific observed object. This is vital for knowledge transfer: if a robot knows how to grasp a "canonical" spray bottle, it can use the deformation field to adapt that grasp to a taller, thinner, or curvier bottle.

Traditionally, this requires high-quality 3D data from RGB-D sensors (like Kinects). However, depth sensors rely on infrared light or structured patterns that fail on transparent surfaces (bottles) or specular surfaces (shiny drills). Furthermore, a single view always leaves the "back" of the object occluded. This paper aims to perform this 3D mapping using only standard 2D color images, leveraging category-level "prior knowledge" to fill in the blanks.

Methods

The proposed pipeline consists of three main stages:

  1. The CNN (Deformation Inference): The authors use a modified FlowNet2 architecture. The network takes four inputs: an observed RGB image, its bounding box mask, a rendered image of the "canonical" (template) model in the same pose, and the template's mask. The CNN predicts a 3D deformation vector for every visible pixel.
  2. The Shape Space: To handle occluded parts, the authors pre-calculate a "latent space" of possible deformations for a category (e.g., "all the ways a drill can vary") using Principal Component Analysis (PCA).
  3. Optimization: The system takes the CNN’s predictions for the visible parts and finds the best-fitting vector within the latent shape space. Because the shape space represents valid category-wide variations, the resulting deformation naturally "guesses" the shape of the occluded parts, ensuring the final 3D model looks like a realistic member of that category.
  4. Zoom Operation: To improve accuracy, the system crops and upsamples the images based on a bounding box before feeding them to the network, ensuring the CNN focuses on fine geometric details.

Datasets

Since no standard dataset for intra-class 3D deformation existed, the authors created their own:

  • Categories: Spray bottles, cameras, bottles, and drills.
  • Source: Realistic 3D textured meshes from online databases (e.g., Sketchfab).
  • Augmentation: To overcome the scarcity of high-quality models, they developed a method to synthetically interpolate between models. By morphing existing meshes, they generated thousands of unique training instances with ground-truth deformation fields.
  • Evaluation: The models were tested on both synthetic data and real-world images of objects, including challenging transparent bottles.

Key Insights and Ideas

  • RGB as a Geometry Proxy: The paper proves that a CNN can learn to map 2D texture and shading cues directly to 3D spatial deformations, bypassing the need for an explicit depth map.
  • Latent Space as a Regularizer: The use of a PCA-based shape space acts as a "sanity check." It prevents the network from producing physically impossible shapes by forcing the output to stay within the bounds of known category variations.
  • Handling "Invisible" Objects: Because the method relies on appearance rather than light reflection/refraction, it successfully registers transparent glass bottles—objects that often appear as "holes" or noise to standard 3D scanners.

Key Results

  • Superiority over CPD: The method significantly outperformed Coherent Point Drift (CPD), a standard non-rigid registration algorithm. For example, on a camera test instance, the proposed method achieved an error of 102.17 µm compared to CPD’s 407.04 µm.
  • Robustness to Pose: The system remains effective even when the initial pose estimation is noisy (tested with ±0.05m/rad noise).
  • Real-World Success: Qualitative results on real-world images show the system successfully reconstructing the 3D geometry of spray bottles and transparent water bottles from a single photo.

Limitations and Future Work

  • Category Dependence: The system requires a pre-trained shape space for each category. It cannot currently handle an "unknown" object type.
  • Initial Pose Requirement: The method assumes a rough initial pose is known (though it is robust to some error). The authors suggest integrating a pose refinement module directly into the network in the future.
  • Masking: The current version uses bounding boxes; the authors plan to move toward pixel-wise semantic segmentation to better handle cluttered environments.

Summary updated

Abstract

In this paper, we propose a novel approach to solve the 3D non-rigid registration problem from RGB images using Convolutional Neural Networks (CNNs). Our objective is to find a deformation field (typically used for transferring knowledge between instances, e.g., grasping skills) that warps a given 3D canonical model into a novel instance observed by a single-view RGB image. This is done by training a CNN that infers a deformation field for the visible parts of the canonical model and by employing a learned shape (latent) space for inferring the deformations of the occluded parts. As result of the registration, the observed model is reconstructed. Because our method does not need depth information, it can register objects that are typically hard to perceive with RGB-D sensors, e.g. with transparent or shiny surfaces. Even without depth data, our approach outperforms the Coherent Point Drift (CPD) registration method for the evaluated object categories.
BibTeX
@inproceedings{rodriguez2020category,
  author = {Diego Rodriguez and Florian Huber and Sven Behnke},
  title = {{Category-Level 3D Non-Rigid Registration from Single-View RGB Images}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=K9KxJ8H6EGTJ},
}

Loading related content...

Figure