Unknown Object Segmentation from Stereo Images

Maximilian Durner, Wout Boerdijk, Martin Sundermeyer, Werner Friedl, Zoltán-Csaba Márton, Rudolph Triebel

TL;DR

This paper introduces INSTR, a novel transformer-based approach for segmenting unknown objects from stereo images. By leveraging subtle viewpoint changes inherent in stereo pairs, INSTR bypasses the need for explicit depth maps, outperforming existing methods that rely on potentially noisy depth data.

What Problem Does This Paper Address?

Autonomous robots operating in dynamic, real-world environments frequently encounter objects they haven't been explicitly trained on. Traditional object instance segmentation methods often require prior knowledge of object categories or 3D models, severely limiting their applicability. While depth information can aid in segmenting unknown objects, affordable depth sensors often produce noisy or incomplete data, especially for challenging surfaces like untextured, metallic, or transparent materials. This paper tackles the problem of Unknown Object Instance Segmentation (UOIS) by developing a method that can robustly identify and segment novel objects without relying on pre-existing semantic knowledge or high-quality depth maps.

Methods

The core of the proposed method, INSTR (Instance Stereo Transformer), is a transformer encoder-decoder architecture adapted from DETR. Key innovations include:

  • Stereo-based Feature Extraction: Instead of relying solely on RGB or depth, INSTR processes a stereo image pair. It uses a ResNet-50 backbone followed by specialized local horizontal correlation layers. These layers are designed to capture stereoscopic information by correlating features from corresponding regions in the left and right images.
  • Sub-pixel Sampling for Correlation: A crucial aspect is the introduction of a sub-pixel sampling mechanism within the correlation layers. This allows the network to dynamically adapt to varying camera intrinsics (like stereo baseline and focal length) without retraining, making it more versatile for different sensor setups. This is achieved by relating the correlation layer's displacement step to estimated camera parameters and using bilinear grid sampling.
  • Direct Instance Prediction: The transformer decoder is modified to directly predict 2D object instance masks. This bypasses intermediate detection steps, leading to a more streamlined and efficient pipeline.
  • Auxiliary Disparity Prediction: To guide the network in effectively utilizing stereo cues, an auxiliary decoder is employed to predict a dense disparity map. This map serves as a training signal, improving the overall segmentation performance.
  • Permutation-Invariant Loss: A modified loss function, incorporating an exponential logarithmic dice loss, is used to handle the challenge of predicting complete zero masks for background and to better focus on difficult segmentation cases.

Datasets

  • Synthetic Training Data: The authors generated a large synthetic dataset using BlenderProc, combining ShapeNet 3D models with procedural data generation on various table-top surfaces. This dataset consists of 40,000 images.
  • STIOS (Stereo Instances On Surfaces) Dataset: A novel dataset introduced by the authors, consisting of 192 stereo images captured with two different stereo sensors (rc_visard and Zed) on eight different tabletop surfaces. It features manually labeled pixel-wise instance segmentation masks for objects in both simple and difficult (physically touching or stacked) configurations.
  • OCID Dataset: Used for additional evaluation of the single-RGB INSTR variant.

Key Insights and Ideas

  • Stereo Viewpoint Changes as a Primary Cue: The paper demonstrates that subtle viewpoint differences in stereo image pairs are often sufficient to infer object boundaries, even without explicit depth information. This challenges the common reliance on noisy depth maps.
  • Dynamic Adaptation to Sensor Intrinsics: The sub-pixel sampling in the correlation layers is a key enabler for generalizing to unseen camera parameters, making the method more robust in real-world robotic applications where sensor configurations can vary.
  • Joint Learning of Disparity and Segmentation: The auxiliary disparity prediction task acts as a powerful regularizer, guiding the network to better exploit stereo cues for improved instance segmentation.
  • Transformer's Suitability for Set Prediction: The transformer architecture's inherent ability to handle set-prediction problems makes it well-suited for directly outputting instance masks without complex post-processing.

Key Results

  • State-of-the-Art Performance: INSTR outperforms existing stereo-based and depth-based UOIS methods on the STIOS dataset, achieving competitive results even with a Mask R-CNN trained on seen objects.
  • Efficiency: The proposed method achieves an inference speed of approximately 18 frames per second on an Nvidia RTX 2080, making it suitable for real-time robotic applications.
  • Generalization: The method shows promising generalization capabilities to unseen objects and different domains, as demonstrated by qualitative results on arbitrary objects.
  • Robustness to Sensor Variations: Evaluation on test scenes with untrained sensor intrinsics (rc_visard and Zed) confirms the effectiveness of the sub-pixel sampling mechanism in adapting to different camera parameters.

Limitations and Future Work

  • Failure Cases: The paper acknowledges specific failure cases, such as when an object gets separated into multiple parts (e.g., the Micky Mouse example) or when separating objects in particularly challenging scenarios.
  • Training Data: While synthetic data is effective, the authors suggest that more diverse training data could further improve performance.
  • Future Work: The authors implicitly suggest exploring more complex scenes and object types, as well as further refining the transformer architecture for even greater efficiency and accuracy. The ability to adapt to untrained sensor intrinsics opens avenues for deployment on a wider range of robotic platforms.

Summary updated

Abstract

Although instance-aware perception is a key prerequisite for many autonomous robotic applications, most of the methods only partially solve the problem by focusing solely on known object categories. However, for robots interacting in dynamic and cluttered environments, this is not realistic and severely limits the range of potential applications. Therefore, we propose a novel object instance segmentation approach that does not require any semantic or geometric information of the objects beforehand. In contrast to existing works, we do not explicitly use depth data as input, but rely on the insight that slight viewpoint changes, which for example are provided by stereo image pairs, are often sufficient to determine object boundaries and thus to segment objects. Focusing on the versatility of stereo sensors, we employ a transformer-based architecture that maps directly from the pair of input images to the object instances. This has the major advantage that instead of a noisy, and potentially incomplete depth map as an input, on which the segmentation is computed, we use the original image pair to infer the object instances and a dense depth map. In experiments in several different application domains, we show that our Instance Stereo Transformer (INSTR) algorithm outperforms current state-of-the-art methods that are based on depth maps. Training code and pretrained models are available at https://github.com/DLR-RM/instr.
BibTeX
@inproceedings{durner2021unknown,
  author = {Maximilian Durner and Wout Boerdijk and Martin Sundermeyer and Werner Friedl and Zoltán-Csaba Márton and Rudolph Triebel},
  title = {{Unknown Object Segmentation from Stereo Images}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=IbRJLDzdhE},
}
Figure