SE3-Nets: Learning Rigid Body Motion using Deep Neural Networks

Arunkumar Byravan, Dieter Fox

Abstract

We introduce SE3-Nets which are deep neural networks designed to model and learn rigid body motion from raw point cloud data. Based only on sequences of depth images along with action vectors and point wise data associations, SE3-Nets learn to segment effected object parts and predict their motion resulting from the applied force. Rather than learning point wise flow vectors, SE3-Nets predict SE(3) transformations for different parts of the scene. Using simulated depth data of a table top scene and a robot manipulator, we show that the structure underlying SE3-Nets enables them to generate a far more consistent prediction of object motion than traditional flow based networks. Additional experiments with a depth camera observing a Baxter robot pushing objects on a table show that SE3-Nets also work well on real data.

Introduction

For a robot to successfully interact with its environment—such as pushing a box across a table—it must be able to predict the physical outcomes of its actions. Traditionally, this predictive capability relies on physics engines like MuJoCo. However, these engines require precise physical parameters (mass, friction, center of gravity) that are notoriously difficult to estimate from visual sensors alone.

Deep learning offers an alternative, but standard approaches often attempt to predict unstructured "optical flow" (3D displacement for every point). Because these models lack an inherent understanding of object rigidity, their predictions frequently result in "smears" or "ghosts," where objects unnaturally deform or dissolve over time. SE3-Nets bridge this gap by combining the flexibility of deep learning with the structural constraints of rigid-body physics, enforcing physical consistency to produce sharper, more accurate environmental predictions.

Methods

The core innovation of SE3-Nets is the representation of motion through $SE(3)$ transformations rather than individual point velocities. The network decomposes a scene into $k$ potentially moving parts and predicts two distinct outputs:

  1. $k$ Motion Masks: Dense, point-wise masks that segment the scene into distinct rigid components (e.g., a ball, a box, and the background).
  2. $k$ $SE(3)$ Transforms: A set of 3D rotations and translations—one for each identified mask.
The SE3-Net architecture: An encoder-decoder structure that predicts motion masks and SE(3) transformations by fusing 3D input with action vectors.

As shown in Figure 1, the architecture utilizes an Encoder to fuse the 3D point cloud with the robot's action vector. The Decoder then outputs the masks and transforms. A final differentiable Transform Layer applies the predicted $SE(3)$ transforms to the input points, weighted by the masks, to generate the predicted future state. To ensure the network learns truly rigid parts, a weight sharpening technique is employed during training, forcing masks toward binary values (0 or 1) and preventing the network from blending multiple transforms into non-physical motions.

Results and Evaluation

The authors evaluated SE3-Nets across simulated environments (Gazebo) and real-world robot interactions. Simulated tasks included a ball colliding with boxes and a Baxter robot arm moving its joints. Real-world tests involved a physical Baxter robot poking household items like Cheez-It boxes and Pringles cans.

Superior Accuracy and Consistency

In the "Single Box" task, SE3-Nets achieved a Mean Squared Error (MSE) of 1.65 cm, significantly outperforming the 2.48 cm MSE of standard flow-based networks. More importantly, SE3-Nets demonstrated superior temporal consistency. When predictions were fed back into the network to simulate multiple steps into the future, SE3-Nets maintained sharp object boundaries, whereas flow-based baselines quickly degraded into unrecognizable point clouds.

Qualitative comparison between SE3-Nets (Ours) and flow-based baselines. SE3-Nets maintain sharp object boundaries while flow-based models produce 'smeared' predictions.

Emergent Segmentation

A key insight of this work is emergent segmentation. The network learns to segment objects (e.g., distinguishing a ball from a box) without any ground-truth object labels. It discovers these segments because decomposing the scene into rigid parts is the most efficient way to minimize prediction error under the $SE(3)$ constraint. This constraint acts as a powerful regularizer, making the model robust to depth sensor noise and data association errors.

Discussion and Future Work

While SE3-Nets show significant promise in learning physical intuition, some limitations remain. The current model requires a pre-defined number of moving parts ($k$) and assumes known point-wise data associations during training. Future iterations could utilize Iterative Closest Point (ICP) matching to remove the association requirement and dynamically determine the number of objects in a scene. Additionally, while the model is optimized for rigidity, extending this framework to handle non-rigid bodies like cloth or liquids remains an open challenge.

Summary updated

Abstract

We introduce SE3-Nets, which are deep neural networks designed to model and learn rigid body motion from raw point cloud data. Based only on sequences of depth images along with action vectors and point wise data associations, SE3-Nets learn to segment effected object parts and predict their motion resulting from the applied force. Rather than learning point wise flow vectors, SE3-Nets predict SE3 transformations for different parts of the scene. Using simulated depth data of a table top scene and a robot manipulator, we show that the structure underlying SE3-Nets enables them to generate a far more consistent prediction of object motion than traditional flow based networks. Additional experiments with a depth camera observing a Baxter robot pushing objects on a table show that SE3-Nets also work well on real data.
BibTeX
@inproceedings{byravan2017se,
  author = {Arunkumar Byravan and Dieter Fox},
  title = {{SE3-Nets: Learning Rigid Body Motion using Deep Neural Networks}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2017},
  url = {https://openreview.net/forum?id=81QN8dList},
}
Figure