Transferring Implicit Knowledge of Non-Visual Object Properties Across Heterogeneous Robot Morphologies

Gyan Tatiya, Jonathan Francis, Jivko Sinapov

Abstract Humans leverage multiple sensor modalities when interacting with objects and discovering their intrinsic properties. Using the visual modality alone is insufficient for deriving intuition behind object properties (e.g., which of two boxes is heavier), making it essential to consider non-visual modalities as well, such as the tactile and auditory. Whereas robots may leverage various modalities to obtain object property understanding via learned exploratory interactions with objects (e.g., grasping, lifting, and shaking behaviors), challenges remain: the implicit knowledge acquired by one robot via object exploration cannot be directly leveraged by another robot with different morphology, because the sensor models, observed data distributions, and interaction capabilities are different across these different robot configurations. To avoid the costly process of learning interactive object perception tasks from scratch, we propose a multi-stage projection framework for each new robot for transferring implicit knowledge of object properties across heterogeneous robot morphologies. We evaluate our approach on the object-property recognition and object-identity recognition tasks, using a dataset containing two heterogeneous robots that perform 7,600 object interactions. Results indicate that knowledge can be transferred across robots, such that a newly-deployed robot can bootstrap its recognition models without exhaustively exploring all objects. We also propose a data augmentation technique and show that this technique improves the generalization of models.

1. Introduction

Interactive object perception—learning about objects through physical interactions like touching, grasping, and shaking—is a data-intensive process. Currently, deploying a new robot model requires it to undergo a lengthy "sensory education" from scratch. This is because sensory data is embodiment-specific; the "language" of sensors on a Baxter robot differs significantly from those on a UR5 arm, even when performing identical tasks. As shown in Figure 1, the audio, effort, and force features recorded during a "shake" behavior vary greatly across morphologies, making direct model transfer impossible.

Figure 1: Comparison of sensory features (Audio, Effort, Force) across Baxter and UR5 robots, illustrating the heterogeneity of data distributions for the same 'Shake' behavior.

2. Methodology

We propose a multi-stage projection framework to bridge the gap between a source (experienced) robot and a target (newly deployed) robot. We evaluated two primary mathematical approaches:

  1. Encoder-Decoder Network (EDN): An encoder maps source features to a latent code, which a decoder then reconstructs into the target robot's feature space.
  2. Kernel Manifold Alignment (KEMA): This approach projects features from both robots into a shared latent space where similar object properties cluster together, regardless of the robot's morphology.

The framework utilizes two types of correspondences: object-identity (mapping interactions with the same physical object) and object-property (mapping interactions with different objects sharing a property, like weight). Additionally, a data augmentation technique was implemented, sampling new trials based on the mean and standard deviation of existing data to prevent overfitting.

Figure 2: Overview of the transfer framework (A, B), the sequence of exploratory behaviors (C), and the object dataset (D).

3. Experimental Evaluation

The study utilized a real-world dataset involving a 7-DOF Baxter and a 6-DOF UR5 robot. The robots interacted with 95 cylindrical containers varying in color, content (e.g., rice, marbles, nuts & bolts), and weight. A total of 7,600 interactions were recorded across eight behaviors: Look, Grasp, Pick, Hold, Shake, Lower, Drop, and Push.

4. Results and Discussion

  • KEMA Performance: The shared latent space approach (KEMA) consistently outperformed the direct translation approach (EDN). This suggests that finding universal representations of object properties is more effective than attempting to simulate one robot's specific sensors using another's.
  • Bootstrapping: In identity recognition tasks, the target robot often performed better using projected features from the source robot than it did using its own limited raw data. For instance, KEMA achieved a mean accuracy delta ($m\Delta A$) of -40.67 for the UR5 robot, indicating a substantial reduction in error rates.
  • Efficiency: The framework allows a newly deployed robot to achieve high recognition accuracy with only a fraction of the physical interactions typically required, effectively "skipping" the most time-consuming phase of training.

5. Conclusion

This research demonstrates that implicit knowledge of non-visual object properties can be successfully transferred across heterogeneous robot morphologies. By leveraging a shared latent feature space, newly deployed robots can bootstrap their understanding of the physical world from the experiences of veteran systems. Future work will explore active learning for autonomous behavior selection and scaling the framework to larger, more diverse robot fleets.

Summary updated

Abstract

Humans leverage multiple sensor modalities when interacting with objects and discovering their intrinsic properties. Using the visual modality alone is insufficient for deriving intuition behind object properties (e.g., which of two boxes is heavier), making it essential to consider non-visual modalities as well, such as the tactile and auditory. Whereas robots may leverage various modalities to obtain object property understanding via learned exploratory interactions with objects (e.g., grasping, lifting, and shaking behaviors), challenges remain: the implicit knowledge acquired by one robot via object exploration cannot be directly leveraged by another robot with different morphology, because the sensor models, observed data distributions, and interaction capabilities are different across these different robot configurations. To avoid the costly process of learning interactive object perception tasks from scratch, we propose a multi-stage projection framework for each new robot for transferring implicit knowledge of object properties across heterogeneous robot morphologies. We evaluate our approach on the object-property recognition and object-identity recognition tasks, using a dataset containing two heterogeneous robots that perform 7,600 object interactions. Results indicate that knowledge can be transferred across robots, such that a newly-deployed robot can bootstrap its recognition models without exhaustively exploring all objects. We also propose a data augmentation technique and show that this technique improves the generalization of models. We release our code and datasets, here: https://github.com/gtatiya/Implicit-Knowledge-Transfer.
BibTeX
@inproceedings{tatiya2023transferring,
  author = {Gyan Tatiya and Jonathan Francis and Jivko Sinapov},
  title = {{Transferring Implicit Knowledge of Non-Visual Object Properties Across Heterogeneous Robot Morphologies}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2023},
  doi = {10.1109/icra48891.2023.10160811},
}
Figure