Kinematic Morphing Networks for Manipulation Skill Transfer

Peter Englert, Marc Toussaint

TL;DR

Researchers have developed Kinematic Morphing Networks (KMN), a deep learning approach that allows robots to transfer manipulation skills (like opening a door) across different environments. By iteratively "morphing" a depth-camera observation into a standardized prototype model, the system extracts precise kinematic parameters that allow a single optimized policy to work on objects of varying shapes, sizes, and positions.

What Problem Does This Paper Address?

In robotics, "skill transfer" is the holy grail: if a robot learns to open one door, it should be able to open any door. However, raw sensor data (like depth images) is high-dimensional, noisy, and viewpoint-dependent. A door in one house might be wider, have a higher handle, or be positioned differently than a door in another.

Standard policies often struggle to bridge this gap because they can't easily map raw pixels to the underlying physical structure of the object. This paper addresses the challenge of extracting a low-dimensional kinematic description (the "skeleton" of the object, including its joints and dimensions) from partial, high-dimensional depth observations to enable seamless skill generalization.

Methods

The authors propose a framework consisting of three main pillars:

  1. Kinematic Morphing: Instead of just identifying an object, the system estimates "morphing parameters." These include transformation parameters ($\theta$)—like 3D rotation and translation—and configuration parameters ($\gamma$)—like the specific height of a door handle or the width of a frame.
  2. Iterative Network Predictions: This is the core innovation. Rather than predicting all parameters in one "shot," the network operates like a controller. It predicts a small transformation, applies it to the observed point cloud, renders a new depth image, and repeats. This iterative process (typically 5 steps) allows the network to "nudge" the observation until it perfectly overlays a standardized prototype model.
  3. Synthetic-to-Real Training: The network is trained entirely on synthetic data generated by an OpenGL renderer. To improve robustness, they use a data augmentation strategy where the network’s own (initially imperfect) predictions are used to create new training samples, effectively fine-tuning the model on "near-miss" scenarios.

Once the parameters are extracted, the robot uses Constrained Trajectory Optimization (KOMO) to calculate the specific movements needed to interact with the environment based on the morphed model.

Datasets

The study primarily uses large-scale synthetic datasets generated in a simulator:

  • Box Scenarios: Three levels of complexity (A, B, and C) with up to 100,000 samples, involving translation, rotation, and scaling.
  • Door Scenario: 100,000 samples featuring variations in door height, width, and handle position.
  • Real-World Data: For validation, the authors used depth data from a Kinect v1 sensor to test the "Box C" task in a physical environment.

Key Insights and Ideas

  • Iterative is Better than One-Step: The "aha moment" is that mapping a raw image to a final parameter set is hard, but mapping an image to a relative change toward a prototype is much easier for a neural network to learn accurately.
  • The Power of the Prototype: By defining a "perfect" prototype door, the robot only needs one optimized policy. The KMN acts as a bridge, translating any weirdly shaped door into the language of that prototype.
  • Differentiable-like Loop: While the renderer itself isn't necessarily differentiable, the loop of "predict $\rightarrow$ transform $\rightarrow$ render" mimics the behavior of iterative registration algorithms like ICP (Iterative Closest Point) but with the added ability to handle non-rigid changes like handle height.

Key Results

  • Superior Accuracy: In the "Box B" task (translation + rotation), KMN achieved a Mean Absolute Error (MAE) of 0.00734, vastly outperforming the non-iterative Baseline (0.06968) and standard ICP (0.62813).
  • Handling Complexity: On the complex "Door" task, KMN maintained high precision across 7 different parameters (3 for pose, 4 for configuration), whereas traditional methods like ICP cannot handle configuration changes (like handle height) at all.
  • Real-World Transfer: The model trained on "fake" simulator data successfully predicted parameters for real boxes observed by a Kinect camera, proving the geometric representation is robust enough to cross the "sim-to-real" gap.
  • Physical Execution: The authors demonstrated a PR2 robot successfully using these extracted parameters to open different doors in a simulated environment.

Limitations and Future Work

  • Known Structure: The system currently requires the robot to know the structure of the object beforehand (e.g., it must know it's looking at a door with one handle). It cannot yet discover the kinematic structure of a completely unknown object.
  • Background Noise: The experiments assumed the background had been removed from the depth data. In messy, real-world kitchens, segmenting the object from the clutter remains a prerequisite step.
  • Configuration Iteration: Currently, only transformation parameters ($\theta$) are refined iteratively; configuration parameters ($\gamma$) are only predicted in the final step. Future work could involve making the entire parameter set part of the iterative loop.

Summary updated

Abstract

The transfer of a robot skill between different geometric environments is non-trivial since a wide variety of environments exists, sensor observations as well as robot motions are high-dimensional, and the environment might only be partially observed. We consider the problem of extracting a low-dimensional description of the manipulated environment in form of a kinematic model. This allows us to transfer a skill by defining a policy on a prototype model and morphing the observed environment to this prototype. A deep neural network is used to map depth image observations of the environment to morphing parameter, which include transformations and configurations of the prototype model. Using the concatenation property of affine transformations and the ability to convert point clouds to depth images allows to apply the network in an iterative manner. The network is trained on data generated in a simulator and on augmented data that is created with its own predictions. The algorithm is evaluated on different tasks, where it is shown that iterative predictions lead to a higher accuracy than one-step predictions.
BibTeX
@inproceedings{englert2018kinematic,
  author = {Peter Englert and Marc Toussaint},
  title = {{Kinematic Morphing Networks for Manipulation Skill Transfer}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2018},
  url = {https://openreview.net/forum?id=nAhDmKjHRh},
}

Loading related content...

Figure