Motion-based Object Segmentation based on Dense RGB-D Scene Flow

Lin Shao, Parth Shah, Vikranth Dwaracherla, Jeannette Bohg

Abstract

Given two consecutive RGB-D images, we propose a model that estimates a dense 3D motion field, also known as scene flow. We take advantage of the fact that in robot manipulation scenarios, scenes often consist of a set of rigidly moving objects. Our model jointly estimates (i) the segmentation of the scene into an unknown but finite number of objects, (ii) the motion trajectories of these objects and (iii) the object scene flow. We employ an hourglass, deep neural network architecture. In the encoding stage, the RGB and depth images undergo spatial compression and correlation. In the decoding stage, the model outputs three images containing a per-pixel estimate of the corresponding object center as well as object translation and rotation. This forms the basis for inferring the object segmentation and final object scene flow. To evaluate our model, we generated a new and challenging, large-scale, synthetic dataset that is specifically targeted at robotic manipulation. In quantitative experiments, we show that we outperform state-of-the-art scene flow and motion-segmentation methods on this data set. In qualitative experiments, we show how our learned model transfers to challenging real-world scenes.

Introduction

For a robot to effectively manipulate its environment, it needs to understand how objects move when interacted with—a concept known as Interactive Perception. A key part of this is estimating scene flow, which is the dense 3D motion field of the environment (the 3D equivalent of optical flow).

Traditionally, estimating scene flow and segmenting moving objects relies on tracking pixels across frames. However, these methods often break down in real-world robotic scenarios due to non-Lambertian surfaces, large displacements, heavy occlusions, and textureless objects. While deep learning has revolutionized 2D optical flow, applying it to 3D scene flow for robotic manipulation—where multiple objects might be colliding, sliding, and occluding one another—remains a significant challenge. Furthermore, existing datasets are mostly geared toward autonomous driving rather than close-up, cluttered manipulation tasks.

Methodology

The proposed architecture is an hourglass-style (encoder-decoder) Convolutional Neural Network that takes two consecutive RGB-D frames as input.

Feature Extraction & Correlation

The network uses two Siamese encoders—one for RGB images and one for point clouds (XYZ data derived from depth). A standout feature is the correlation layer applied to the RGB features. It calculates the similarity between patches in consecutive frames. These RGB correlation values are then used to weight the XYZ point cloud features. Intuitively, this tells the network: "If these pixels look the same in RGB across frames, their 3D depth points probably correspond to the same moving object."

Detailed CNN architecture for joint scene flow and segmentation.

Direct Rigid Motion Prediction

Instead of just predicting raw 3D displacement vectors, the decoder outputs three specific images: a per-pixel estimate of the object's center, its 3D translation, and its 3D rotation. By forcing the network to predict rigid body transformations and object centers rather than just raw 3D flow vectors, the model inherently learns the concept of "objectness." This makes the downstream clustering/segmentation step much cleaner.

Motion-Based Segmentation

Because pixels belonging to the same rigid object will predict the same object center and trajectory, these predictions are used to cluster pixels into distinct object masks. The authors also predict an "enclosing sphere radius" to help bound these clusters. This approach allows the model to segment an unknown but finite number of objects based on their shared motion trajectories.

Clustering process in the trajectory feature space for object segmentation.

Dataset Generation

Because existing datasets either lack dense depth data or don't represent close-up robotic manipulation, the authors built a new large-scale synthetic dataset.

  • Physics-Based Simulation: They used 31,594 3D meshes from ShapeNet and simulated realistic physics (falling, colliding, sliding) using the Bullet physics engine.
  • Realistic Rendering: They rendered 35,540 pairs of consecutive RGB-D frames using Blensor to simulate realistic depth sensor noise.
  • Rotational Symmetry Annotation: A major contribution is how they handled rotationally symmetric objects. If a uniformly colored bottle rotates around its axis, the visual data doesn't change. The authors mathematically mapped all ground-truth rotations to the minimum angular displacement to ensure stable network convergence.

Experimental Results

  • Scene Flow Accuracy: The full model achieved an End Point Error (EPE) of 1.303 cm, significantly outperforming baseline methods like PD-Flow (2.830 cm) and Jaimez et al. (2.330 cm).
  • Segmentation Performance: The model vastly outperformed the state-of-the-art traditional method (HOMC). On a 2-frame test set, the proposed model achieved an F-measure of 0.730, whereas HOMC failed (0.111) due to its reliance on longer video sequences.
  • Speed: The model runs at 8.3 frames per second, making it viable for near real-time robotic perception.
  • Zero-Shot Real-World Transfer: Despite being trained entirely on synthetic data, the model successfully generalized to real-world RGB-D data captured by an Intel RealSense camera, accurately segmenting objects in cluttered scenes without fine-tuning.

Conclusion

This work introduces a robust framework for joint scene flow estimation and object segmentation. By leveraging cross-modal correlation and rigid body priors, the model overcomes common pitfalls in robotic perception. While limitations like boundary noise and the assumption of rigidity remain, the successful zero-shot transfer to real-world data highlights the potential for physics-based synthetic training. Future work will explore domain adaptation and the application of this dense flow to complex manipulation tasks.

Summary updated

Abstract

Given two consecutive RGB-D images, we propose a model that estimates a dense 3D motion field, also known as scene flow. We take advantage of the fact that in robot manipulation scenarios, scenes often consist of a set of rigidly moving objects. Our model jointly estimates (i) the segmentation of the scene into an unknown but finite number of objects, (ii) the motion trajectories of these objects and (iii) the object scene flow. We employ an hourglass, deep neural network architecture. In the encoding stage, the RGB and depth images undergo spatial compression and correlation. In the decoding stage, the model outputs three images containing a per-pixel estimate of the corresponding object center as well as object translation and rotation. This forms the basis for inferring the object segmentation and final object scene flow. To evaluate our model, we generated a new and challenging, large-scale, synthetic dataset that is specifically targeted at robotic manipulation: It contains a large number of scenes with a very diverse set of simultaneously moving 3D objects and is recorded with a simulated, static RGB-D camera. In quantitative experiments, we show that we outperform state-of-the-art scene flow and motion-segmentation methods on this data set. In qualitative experiments, we show how our learned model transfers to challenging real-world scenes, visually generating better results than existing methods.
BibTeX
@article{shao2018motion,
  author = {Lin Shao and Parth Shah and Vikranth Dwaracherla and Jeannette Bohg},
  title = {{Motion-based Object Segmentation based on Dense RGB-D Scene Flow}},
  journal = {IEEE Robotics Autom. Lett.},
  year = {2018},
  url = {https://openreview.net/forum?id=GLu7wozuPOb},
}
Figure