Generate What You Can't See - a View-dependent Image Generation

Karol Piaskowski, Rafal Staszak, Dominik Belter

TL;DR

This paper introduces a novel method for generating multiple RGB and depth views of an object from a single input image. This is achieved using a deep neural network that learns to "imagine" an object's appearance from various perspectives, enabling robots to better understand and interact with unfamiliar objects without exhaustive scanning.

What Problem Does This Paper Address?

Autonomous robots operating in complex environments often struggle with object manipulation due to incomplete knowledge of an object's shape and appearance. Traditional methods require extensive scanning of the workspace, which is time-consuming and sometimes physically impossible due to occlusions or reachability constraints. Humans, however, can infer missing object information from a single view by leveraging past experience and prior knowledge. This paper aims to bridge this gap by enabling robots to perform similar "shape completion" by generating plausible views of an object from a single input image, thereby improving their ability to plan actions like grasping.

Methods

The proposed system consists of two main modules: an object extractor and a generative network.

  1. Object Extractor: This module uses a pre-trained Mask R-CNN to detect objects in a scene, segment them, and extract their bounding boxes. The detected objects are then cropped, resized to a fixed 128x128 resolution, and padded to create square images. Both the RGB image and its corresponding mask are concatenated and fed into the generative network. The class prediction from Mask R-CNN is used to select the appropriate generative network for the object.
  2. Generative Network: The core of the system is a U-Net architecture. It takes the concatenated RGB image and object mask, along with information about the desired view angle (pitch and yaw), as input. The network first encodes the input into a latent representation. This latent representation is then combined with the view angle information and passed through fully connected layers. Finally, a decoder reconstructs the desired RGB and depth images for the specified viewpoint. The network is trained to minimize the mean square error between the generated and ground truth images. To handle different object classes, the authors train 12 independent U-Net models, each specialized for a single object class.

Datasets

The authors train their models on the ShapeNet dataset. This dataset contains approximately 51,000 3D models across 55 object classes. For their experiments, they extracted models from specific categories like birdhouse, bottle, bowl, can, car, chair, faucet, guitar, lamp, microphone, mug, and table. For each class, they rendered up to 300 models at various angles, generating both RGB and depth images. They sampled pitch angles from 0° to 30° (in 10° steps) and yaw angles from -360° to 360° (in 12° steps).

For training the Mask R-CNN object detector, they created a synthetic dataset of cans by embedding 20 different can instances into 1000 randomly chosen background images.

Key Insights and Ideas

  • View-dependent generation for robotic manipulation: The paper argues that generating 2D images (RGB and depth) is more efficient and potentially offers better spatial resolution than generating 3D voxel grids, especially for capturing fine details. This approach directly provides information useful for downstream tasks like grasping.
  • Human-like "imagination" for robots: By learning to generate views from unseen angles, the robot can effectively "hallucinate" occluded parts of an object, mimicking human cognitive abilities.
  • Specialized networks for object classes: The authors found that training a single network for all object classes was challenging. Instead, they opted for multiple specialized networks, each trained for a specific object class, which improved performance and reduced training resources.
  • Interpolation capabilities: The network demonstrates an ability to interpolate between training angles, generating plausible views even for angles not explicitly seen during training. This suggests a degree of generalization beyond simple memorization.
  • Robustness to misclassification: Even when presented with an incorrect object class prediction from Mask R-CNN, the generative network can still produce reasonable images, indicating a degree of robustness in modeling rigid object transformations.

Key Results

  • The proposed method can generate sequences of RGB and depth images of an object from various viewpoints using a single input image.
  • Qualitative results show that the generated images are visually plausible and preserve the object's shape and continuity of rotation.
  • For the testing dataset, the accuracy of generated RGB and depth images was evaluated. For instance, the average accuracy for RGB images across all tested classes was 96.2%, and for depth images, it was 98.16%.
  • The paper highlights that complex objects like cars, faucets, and guitars achieved the best accuracy, while birdhouses showed the lowest accuracy, likely due to significant differences between training and testing instances.
  • The accuracy of generated images decreases with increasing rotational distance from the input viewpoint, but remains at a reasonable level.

Limitations and Future Work

  • Lack of texture generation: The neural network was not trained on textured objects and therefore does not generate textures on the output images.
  • Single network for multiple classes: Training a single network to generate images for diverse object classes proved difficult.
  • Handling of object legs: The generated legs of objects like chairs are often described as bent or vague, indicating a challenge in accurately reconstructing complex articulated parts.

Future work includes:

  • Investigating architectures and training strategies to enable texture generation.
  • Developing a single neural network capable of handling multiple object classes.
  • Using the generated images to reconstruct a full 3D model of the object.
  • Estimating camera motion by comparing generated and current camera images.

Summary updated

BibTeX
@inproceedings{piaskowski2019generate,
  author = {Karol Piaskowski and Rafal Staszak and Dominik Belter},
  title = {{Generate What You Can't See - a View-dependent Image Generation}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2019},
  doi = {10.1109/iros40897.2019.8967616},
}

Loading related content...

Figure