Kinematic Morphing Networks for Manipulation Skill Transfer

Peter Englert, Marc Toussaint

Abstract

The transfer of a robot skill between different geometric environments is non-trivial since a wide variety of environments exists, sensor observations as well as robot motions are high-dimensional, and the environment might only be partially observed. We consider the problem of extracting a low-dimensional description of the manipulated environment in form of a kinematic model. This allows us to transfer a skill by defining a policy on a prototype model and morphing the observed environment to this prototype. A deep neural network is used to map depth image observations of the environment to morphing parameter, which include transformation and configuration parameters of the prototype model. Using the concatenation property of affine transformations and the ability to convert point clouds to depth images allows to apply the network in an iterative manner. The network is trained on data generated in a simulator and on augmented data that is created by using network predictions. The algorithm is evaluated on different tasks, where it is shown that iterative predictions lead to a higher accuracy than one-step predictions.

Introduction

In robotics, "skill transfer" is a significant challenge: a robot that learns to open one door should ideally be able to open any door. However, raw sensor data like depth images are high-dimensional, noisy, and viewpoint-dependent. A door in a new environment might have a different width, handle height, or orientation. Standard policies often struggle to bridge this gap because they cannot easily map raw pixels to the underlying physical structure of the object. Kinematic Morphing Networks (KMN) address this by extracting a low-dimensional kinematic description—a "skeleton" of the object—from partial depth observations to enable seamless skill generalization.

Methodology

The KMN framework consists of three main pillars designed to standardize environmental observations for a single optimized policy:

  1. Kinematic Morphing: The system estimates "morphing parameters" that map an observed object to a standardized prototype. These include transformation parameters ($\theta$), such as 3D rotation and translation, and configuration parameters ($\gamma$), such as the specific height of a door handle or the width of a frame.
Kinematic model of a door showing transformation and configuration parameters.
  1. Iterative Network Predictions: Rather than predicting all parameters in a single step, the network operates iteratively. It predicts a small transformation, applies it to the observed point cloud, renders a new depth image, and repeats. This process (typically 5 steps) allows the network to progressively "nudge" the observation until it aligns with the prototype model.
  2. Synthetic-to-Real Training: The network is trained entirely on synthetic data. To improve robustness, the authors use a data augmentation strategy where the network’s own initially imperfect predictions are used to create new training samples, fine-tuning the model on "near-miss" scenarios.

Once parameters are extracted, the robot uses Constrained Trajectory Optimization (KOMO) to calculate the movements required to interact with the environment based on the morphed model.

Key Results and Evaluation

The study evaluated KMN on synthetic datasets for box and door manipulation, as well as real-world validation using a Kinect v1 sensor.

Comparison of Mean Absolute Error (MAE) on test data between the baseline and KMN across multiple iterative predictions.

A primary finding is that iterative predictions significantly outperform one-step baselines. As shown in the figure above, the Mean Absolute Error (MAE) on test data drops sharply after the first prediction and continues to refine, whereas the baseline exhibits higher error and variance. In the "Box B" task (translation + rotation), KMN achieved an MAE of 0.00734, vastly outperforming the non-iterative baseline (0.06968) and standard Iterative Closest Point (ICP) methods (0.62813). Furthermore, KMN successfully handles configuration changes, such as handle height, which traditional registration methods cannot address.

Conclusion

Kinematic Morphing Networks provide a robust method for translating high-dimensional sensor data into actionable kinematic models. By leveraging iterative refinement and prototype morphing, the system allows robots to generalize manipulation skills across geometrically diverse environments. Future work will focus on discovering kinematic structures of unknown objects and improving performance in cluttered real-world scenes.

Summary updated

Abstract

The transfer of a robot skill between different geometric environments is non-trivial since a wide variety of environments exists, sensor observations as well as robot motions are high-dimensional, and the environment might only be partially observed. We consider the problem of extracting a low-dimensional description of the manipulated environment in form of a kinematic model. This allows us to transfer a skill by defining a policy on a prototype model and morphing the observed environment to this prototype. A deep neural network is used to map depth image observations of the environment to morphing parameter, which include transformation and configuration parameters of the prototype model. Using the concatenation property of affine transformations and the ability to convert point clouds to depth images allows to apply the network in an iterative manner. The network is trained on data generated in a simulator and on augmented data that is created by using network predictions. The algorithm is evaluated on different tasks, where it is shown that iterative predictions lead to a higher accuracy than one-step predictions.
BibTeX
@inproceedings{englert2018kinematic,
  author = {Peter Englert and Marc Toussaint},
  title = {{Kinematic Morphing Networks for Manipulation Skill Transfer}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2018},
  url = {https://openreview.net/forum?id=nAhDmKjHRh},
}
Figure