Motion-Based Object Segmentation Based on Dense RGB-D Scene Flow

Lin Shao, Parth Shah, Vikranth Dwaracherla, Jeannette Bohg

TL;DR

This paper introduces a novel deep learning architecture that jointly estimates 3D scene flow and segments an unknown number of rigidly moving objects from just two consecutive RGB-D images. By leveraging a custom correlation layer that uses RGB similarities to weight depth features, and by training on a newly generated physics-based synthetic dataset, the model achieves state-of-the-art performance and successfully transfers to real-world robotic manipulation scenarios without fine-tuning.

What Problem Does This Paper Address?

For a robot to effectively manipulate its environment, it needs to understand how objects move when interacted with—a concept known as Interactive Perception. A key part of this is estimating scene flow, which is the dense 3D motion field of the environment (the 3D equivalent of optical flow).

Traditionally, estimating scene flow and segmenting moving objects relies on tracking pixels across frames. However, these methods often break down in real-world robotic scenarios due to non-Lambertian surfaces, large displacements, heavy occlusions, and textureless objects. While deep learning has revolutionized 2D optical flow, applying it to 3D scene flow for robotic manipulation—where multiple objects might be colliding, sliding, and occluding one another—remains a significant challenge. Furthermore, existing datasets are mostly geared toward autonomous driving (like KITTI) rather than close-up, cluttered manipulation tasks.

Methods

The authors propose an hourglass-style (encoder-decoder) Convolutional Neural Network that takes two consecutive RGB-D frames as input.

  • Feature Extraction & Correlation: The network uses two Siamese encoders—one for RGB images and one for point clouds (XYZ data derived from depth). A standout feature is the correlation layer applied to the RGB features. It calculates the similarity between patches in consecutive frames. These RGB correlation values are then used to weight the XYZ point cloud features. Intuitively, this tells the network: "If these pixels look the same in RGB across frames, their 3D depth points probably correspond to the same moving object."
  • Direct Rigid Motion Prediction: Instead of just predicting raw 3D displacement vectors, the decoder outputs three specific images: a per-pixel estimate of the object's center, its 3D translation, and its 3D rotation.
  • Motion-Based Segmentation: Because pixels belonging to the same rigid object will predict the same object center and trajectory, the authors use these predictions to cluster pixels into distinct object masks. They also predict an "enclosing sphere radius" to help bound these clusters.
  • Custom Losses: The network is trained using a combination of mask loss, pixel-wise regression losses, a variance loss (to force pixels of the same object to predict similar trajectories), and a violation loss (to penalize wild outliers).

Datasets

Because existing datasets (like KITTI or FlyingThings3D) either lack dense depth data or don't represent close-up robotic manipulation, the authors built a new large-scale synthetic dataset.

  • They used 31,594 3D meshes from ShapeNet and simulated realistic physics (falling, colliding, sliding) using the Bullet physics engine.
  • They rendered 35,540 pairs of consecutive RGB-D frames using Blensor to simulate realistic depth sensor noise.
  • Rotational Symmetry Annotation: A major contribution to the dataset is how they handled rotationally symmetric objects (like bottles or bowls). If a uniformly colored bottle rotates around its axis, the visual data doesn't change, which would normally confuse a neural network during training (multiple valid ground truths). The authors mathematically mapped all ground-truth rotations to the minimum angular displacement to ensure stable network convergence.

Key Insights and Ideas

  • Cross-Modal Correlation is Powerful: Relying solely on depth data for tracking is hard when objects slide against each other. By using RGB correlation to guide the depth feature matching, the network becomes highly robust to occlusions and large displacements.
  • Predicting Transforms > Predicting Vectors: By forcing the network to predict rigid body transformations (translation and rotation) and object centers rather than just raw 3D flow vectors, the model inherently learns the concept of "objectness." This makes the downstream clustering/segmentation step much cleaner.
  • The Symmetry Trap: The authors' realization that rotationally symmetric objects create infinite valid solutions for scene flow is a fantastic "aha" moment. Correcting the ground truth to only penalize the minimum observable angular displacement is a clever, necessary trick for training physics-based ML models.

Key Results

  • Scene Flow Accuracy: The full model (OurC+vL+Rig) achieved an End Point Error (EPE) of 1.303 cm, significantly outperforming baseline methods like PD-Flow (2.830 cm) and Jaimez et al. (2.330 cm).
  • Segmentation Performance: The model vastly outperformed the state-of-the-art traditional method (HOMC). On a 2-frame test set, the proposed model achieved an F-measure of 0.730, whereas HOMC completely failed (0.111) because it relies on longer video sequences to build trajectories.
  • Speed: The most complex version of the model runs at 8.3 frames per second (0.121 seconds per frame pair), making it viable for near real-time robotic perception.
  • Zero-Shot Real-World Transfer: Despite being trained entirely on synthetic data, the model successfully generalized to real-world RGB-D data captured by an Intel RealSense SR300 camera, accurately segmenting objects and predicting flow in cluttered, physical scenes.

Limitations and Future Work

The authors acknowledge a few limitations:

  • Boundary Noise: The model sometimes produces inaccurate object boundaries due to noisy real-world sensor data or false positive segmentations caused by varied lighting.
  • Similar Trajectories: If two distinct objects are moving along nearly identical trajectories, the clustering algorithm struggles to segment them into separate entities.
  • Rigidity Assumption: The method inherently assumes objects are rigid and scenes are flat; it cannot generalize to non-rigid objects (like cloth or cables).

For future work, the authors suggest exploring how this dense scene flow can enable agile robotic manipulation in cluttered environments, and utilizing transfer learning (like domain adaptation) to further close the sim-to-real gap.

Summary updated

Abstract

Given two consecutive RGB-D images, we propose a model that estimates a dense three-dimensional (3D) motion field, also known as scene flow. We take advantage of the fact that in robot manipulation scenarios, scenes often consist of a set of rigidly moving objects. Our model jointly estimates the following: First, the segmentation of the scene into an unknown but finite number of objects, second, the motion trajectories of these objects, and finally, the object scene flow. We employ an hourglass, deep neural network architecture. In the encoding stage, the RGB and depth images undergo spatial compression and correlation. In the decoding stage, the model outputs three images containing a per-pixel estimate of the corresponding object center as well as object translation and rotation. This forms the basis for inferring the object segmentation and final object scene flow. To evaluate our model, we generated a new and challenging, large scale, synthetic dataset that is specifically targeted at robotic manipulation: It contains a large number of scenes with a very diverse set of simultaneously moving 3D objects and is recorded with a simulated, static RGB-D camera. In quantitative experiments, we show that we outperform state-of-the-art scene flow and motion-segmentation methods on this data set. In qualitative experiments, we show how our learned model transfers to challenging real-world scenes, visually generating better results than existing methods.
BibTeX
@article{shao2018motion,
  author = {Lin Shao and Parth Shah and Vikranth Dwaracherla and Jeannette Bohg},
  title = {{Motion-Based Object Segmentation Based on Dense RGB-D Scene Flow}},
  journal = {IEEE Robotics Autom. Lett.},
  year = {2018},
  url = {https://openreview.net/forum?id=GLu7wozuPOb},
}

Loading related content...

Figure