Robot Body Schema Learning from Full-body Extero/Proprioception Sensors

Shuo Jiang, Jinkun Zhang, Lawson L. S. WongarXiv 2024 · OpenReview

TL;DR

This paper introduces a novel method for robots to autonomously learn their own body structure (topology) using only on-body exteroception (like IMUs) and proprioception (joint encoders) data. The core contribution is a "Heterogeneous Dependency Matrix" that, when learned, can be translated into a "Heterogeneous Out-tree" representing the robot's body schema, even with noisy or incomplete sensor data. This is crucial for robots operating in dynamic environments or those with unknown or changing morphologies.

What Problem Does This Paper Address?

Traditionally, a robot's body structure (its kinematic tree) is hardcoded by its designers. However, this is a significant limitation for robots that need to adapt to damage, reconfigurations, or operate in environments where their exact structure isn't known beforehand. Humans, on the other hand, possess an innate "body schema" – an internal representation of their body's structure and how its parts relate. This paper aims to equip robots with a similar capability, enabling them to learn their own body topology autonomously from sensor data, much like how humans learn their body schema. The challenge lies in inferring this complex topological information from noisy, potentially incomplete, and indirect sensor readings.

Methods

The authors propose a pipeline that starts with learning the forward kinematics (FK) of each sensor on the robot. This is achieved using a SE(3) neural network that maps joint angles to a homogeneous transformation matrix representing the sensor's pose. The key insight is that the Jacobian of this FK mapping, when made reference-frame independent, encodes the dependency of the sensor's pose on each joint. This "Transform Invariant Jacobian" is then binarized into a "Dependency Feature" for each sensor. Stacking these features from all sensors forms a "Heterogeneous Dependency Matrix." This matrix is then converted into a "Heterogeneous Out-tree," which represents the robot's body topology. The paper also introduces methods to handle partial observability (missing sensors) and noisy data by employing techniques like variance filtering, max functions, and Dirichlet Process Gaussian Mixture Models (DPGMM) for matrix completion and correction.

Datasets

The paper describes experiments conducted on 6 different robots with varying body structures in simulation, and one real UR5e robot. The data was generated by performing random motions on the robot joints and recording exteroception (IMU) and proprioception (joint encoder) data. No specific public datasets are mentioned; the data appears to be generated for the experiments.

Key Insights and Ideas

  • Body Schema for Robots: The paper frames robot body structure learning as analogous to human body schema acquisition, highlighting the importance of self-awareness for robots.
  • Heterogeneous Dependency Matrix: This novel matrix representation effectively captures the topological relationships between joints and sensors, serving as a bridge between raw sensor data and the robot's body structure.
  • Out-tree as Topology: The translation of the dependency matrix into a heterogeneous out-tree provides a clear and interpretable representation of the robot's body topology.
  • Data-Driven Inference: The reliance on on-body sensors (IMUs and joint encoders) for learning, rather than external vision or pre-programmed models, makes the approach more generalizable and robust.
  • Robustness to Noise and Partial Observability: The proposed methods for matrix completion and correction are crucial for real-world applications where sensor data is imperfect.

Key Results

The algorithm successfully identified the ground-truth body structures of 6 simulated robots with different topologies and one real UR5e robot. The authors claim their method outperforms a baseline approach ([21]) in accurately recognizing robot topology, especially when using only IMU data, where the baseline struggles to cluster sensors and links correctly. The paper also demonstrates the effectiveness of their neural network structure and the robustness of their approach to noisy data and partial observability.

Limitations and Future Work

The authors acknowledge that the matrix completion process can be non-unique when multiple sensors are missing. They suggest that uniqueness can be achieved when only one link is unobservable. Future work includes extending the algorithm to locomotion robots and optimizing the training time for faster adaptation. The paper also mentions the potential for this framework to identify the forward kinematics of robots with unknown or hard-to-model structures, such as soft robots.

Summary updated

Abstract

For a robot, its body structure is an a-prior knowledge when it is designed. However, when such information is not available, can a robot recognize it by itself? In this paper, we aim to grant a robot such ability to learn its body structure from exteroception and proprioception data collected from on-body sensors. By a novel machine learning method, the robot can learn a binary Heterogeneous Dependency Matrix from its sensor readings. We showed such matrix is equivalent to a Heterogeneous out-tree structure which can uniquely represent the robot body topology. We explored the properties of such matrix and the out-tree, and proposed a remedy to fix them when they are contaminated by partial observability or data noise. We ran our algorithm on 6 different robots with different body structures in simulation and 1 real robot. Our algorithm correctly recognized their body structures with only on-body sensor readings but no topology prior knowledge.
BibTeX
@misc{jiang2024robot,
  author = {Shuo Jiang and Jinkun Zhang and Lawson L. S. Wong},
  title = {{Robot Body Schema Learning from Full-body Extero/Proprioception Sensors}},
  year = {2024},
  eprint = {2402.18675},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2402.18675},
}

Loading related content...

Figure