Transferring Implicit Knowledge of Non-Visual Object Properties Across Heterogeneous Robot Morphologies

Gyan Tatiya, Jonathan Francis, Jivko Sinapov

Imagine you’re trying to figure out if a sealed opaque container is full of rice or pasta. You wouldn't just look at it; you’d pick it up, shake it, and listen to the sound. Robots can do this too, but there’s a catch: every robot "feels" and "hears" differently based on its specific motors, sensors, and arm structure (its morphology).

In a new paper, researchers from Tufts University and Bosch AI present a framework that allows a "newbie" robot to bootstrap its understanding of object properties by leveraging the "experience" of a veteran robot, even if their physical builds are completely different.

TL;DR

This paper introduces a framework to transfer implicit knowledge of non-visual object properties (like weight and content) between heterogeneous robots. By using manifold alignment and encoder-decoder networks, a newly deployed robot can learn to identify objects and their properties significantly faster than learning from scratch.

What Problem Does This Paper Address?

Interactive object perception—learning about objects by touching, grasping, and shaking them—is a slow, data-hungry process. Currently, if you deploy a new robot model, it usually has to start its "sensory education" from zero.

The challenge is that sensory data is embodiment-specific. A force sensor on a Baxter robot’s gripper produces different data distributions than a similar sensor on a UR5 arm. Because the "language" of their sensors is different, you can't simply copy-paste a machine learning model from one to the other. This paper tackles the "cross-robot transfer" problem: how do we translate the "feel" of an object from Robot A to Robot B?

Methods

The authors propose a multi-stage projection framework to bridge the gap between a source robot (experienced) and a target robot (newly deployed). They compare two primary mathematical approaches:

  1. Encoder-Decoder Network (EDN): A "translation" approach. An encoder turns the source robot's features into a latent code, and a decoder attempts to "reconstruct" what those features would look like if they had been recorded by the target robot.
  2. Kernel Manifold Alignment (KEMA): A "common language" approach. Instead of translating one to the other, KEMA projects features from both robots into a shared, latent mathematical space where similar object properties (e.g., "heavy" or "contains marbles") cluster together regardless of which robot did the shaking.

They also explore two types of correspondences:

  • Object-identity: Mapping interactions with the exact same physical object.
  • Object-property: Mapping interactions with different objects that share a property (e.g., two different red bowls that both weigh 100g).

Finally, they introduce a data augmentation technique that samples additional trials based on the mean and standard deviation of existing data to prevent overfitting and improve generalization.

Datasets

The study uses a substantial real-world dataset (which the authors have released):

  • Robots: A 7-DOF Baxter and a 6-DOF UR5.
  • Objects: 95 cylindrical containers varying in color (5), content (6 types like rice, marbles, or nuts & bolts), and weight (4 levels).
  • Interactions: 7,600 total interactions across 8 behaviors (grasp, shake, push, drop, etc.).
  • Sensors: Audio (microphones), Effort (joint torque), and Force (end-effector sensors).

Key Insights and Ideas

  • Shared Spaces Win: KEMA (the shared latent space) consistently outperformed EDN (direct translation). This suggests that finding a "universal" representation of an object property is more effective than trying to simulate one robot's sensors using another's.
  • Identity over Property: Building correspondences based on specific object identities (Robot A and B both touched "Object #42") creates a much better alignment than just matching general properties (Robot A touched "a heavy object" and Robot B touched "a different heavy object").
  • Better than "Self-Knowledge": In some identity recognition tasks, the target robot actually performed better using projected features from the source robot than it did using its own limited raw data. This indicates that the transferred knowledge provides a cleaner, more generalized signal.

Key Results

  • Property Recognition: In weight and content recognition, both transfer methods significantly outperformed the baseline (learning from scratch) when the target robot had limited experience (interacting with only a few objects).
  • Identity Recognition: KEMA achieved a mean accuracy delta ($m\Delta A$) of -40.67 for the UR5 robot. In this context, a negative number is excellent—it means the error rate dropped significantly compared to the baseline.
  • Efficiency: The framework allows a robot to achieve high recognition accuracy with only a fraction of the usual required physical interactions, effectively "skipping" the most time-consuming part of robot training.

Limitations and Future Work

While the results are promising, the authors note a few hurdles:

  • Shared Context: The current model assumes both robots are performing the same behaviors in the same environment. Future work could explore transferring knowledge when robots use different exploratory behaviors.
  • Autonomous Selection: Currently, the robots perform a fixed sequence of behaviors. The authors want to move toward "active learning," where the robot autonomously chooses which behavior (e.g., "should I shake this or drop it?") would most quickly resolve its uncertainty.
  • Scaling Up: The study focused on two robots. The ultimate goal is a "fleet learning" scenario where dozens of different robot morphologies contribute to a massive, shared library of object property knowledge.

Summary updated

Abstract

Humans leverage multiple sensor modalities when interacting with objects and discovering their intrinsic properties. Using the visual modality alone is insufficient for deriving intuition behind object properties (e.g., which of two boxes is heavier), making it essential to consider non-visual modalities as well, such as the tactile and auditory. Whereas robots may leverage various modalities to obtain object property understanding via learned exploratory interactions with objects (e.g., grasping, lifting, and shaking behaviors), challenges remain: the implicit knowledge acquired by one robot via object exploration cannot be directly leveraged by another robot with different morphology, because the sensor models, observed data distributions, and interaction capabilities are different across these different robot configurations. To avoid the costly process of learning interactive object perception tasks from scratch, we propose a multi-stage projection framework for each new robot for transferring implicit knowledge of object properties across heterogeneous robot morphologies. We evaluate our approach on the object-property recognition and object-identity recognition tasks, using a dataset containing two heterogeneous robots that perform 7,600 object interactions. Results indicate that knowledge can be transferred across robots, such that a newly-deployed robot can bootstrap its recognition models without exhaustively exploring all objects. We also propose a data augmentation technique and show that this technique improves the generalization of models. We release our code and datasets, here: https://github.com/gtatiya/Implicit-Knowledge-Transfer.
BibTeX
@inproceedings{tatiya2023transferring,
  author = {Gyan Tatiya and Jonathan Francis and Jivko Sinapov},
  title = {{Transferring Implicit Knowledge of Non-Visual Object Properties Across Heterogeneous Robot Morphologies}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2023},
  doi = {10.1109/icra48891.2023.10160811},
}

Loading related content...

Figure