A Framework for Sensorimotor Cross-Perception and Cross-Behavior Knowledge Transfer for Object Categorization

Gyan Tatiya, Ramtin Hosseini, Michael C. Hughes, Jivko SinapovFrontiers Robotics AI 2020 · DOI

TL;DR

This paper introduces a novel framework that allows robots to transfer learned knowledge about objects between different sensory modalities (like vision and touch) and different actions (like grasping and lifting). This is crucial for building more adaptable robots that can learn from limited data and recover from sensor failures, ultimately making them more robust and efficient.

What Problem Does This Paper Address?

Robots learn about the world through interaction, much like humans. This involves performing actions (behaviors) and observing the resulting sensory feedback (e.g., what an object feels like, sounds like, or looks like). However, the knowledge a robot gains is often tied to its specific sensors and actions. If a robot has a particular set of sensors and performs certain movements, the learned object models are not directly transferable to another robot with different sensors or actions. This is a significant hurdle for deploying robots in diverse environments. Furthermore, if a robot's sensor fails, it can lose critical information, rendering its learned models useless. This paper tackles these limitations by proposing a method to transfer this "sensorimotor" knowledge, enabling robots to leverage experiences from other robots or other sensory inputs.

Methods

The core of the proposed solution lies in using variational auto-encoders (VAEs) and encoder-decoder networks. These deep learning architectures are adept at learning compressed representations of data and reconstructing it.

The framework proposes two main transfer scenarios:

  1. Cross-Perception Transfer: This allows knowledge to be transferred between different sensory modalities for the same behavior. For example, if a robot's haptic sensor fails, it could use its vision or audio data from the same action to "reconstruct" the missing haptic information.
  2. Cross-Behavior Transfer: This enables knowledge transfer between different behaviors, even if the sensory modalities are also different. This is useful when a new robot needs to learn from a more experienced robot that has explored objects with different actions.

The authors specifically leverage β-Variational Encoder-Decoder (β-VED) and β-Variational Autoencoder (β-VAE) models. These models learn a probabilistic mapping between the feature spaces of different sensorimotor contexts (a combination of behavior and modality). The β-VED is used for mapping from one context to another (e.g., vision of "push" to haptic of "push"), while the β-VAE is more flexible and can handle multiple source contexts to predict a target context. The key idea is to learn a latent representation that captures the essential object properties, allowing for translation between different sensory and behavioral experiences.

Datasets

The primary dataset used in this research is a publicly available dataset introduced by Sinapov et al. (2014a). This dataset features:

  • 100 objects across 20 categories.
  • A robot performing 9 different behaviors (press, grasp, hold, lift, drop, poke, push, shake, tap).
  • Each behavior was performed 5 times on each object.
  • Sensory data collected includes audio, haptic, vibrotactile, and visual feedback.

A second publicly available dataset from Sinapov et al. (2016) was also used for validation. This dataset includes:

  • 32 objects.
  • 8 behaviors (grasp, lift, hold, look, lower, drop, push, press).
  • Haptic and auditory sensory modalities, along with visual data (color histogram, fpfh, and deep visual features).

Key Insights and Ideas

  • Robots can learn to "imagine" sensory experiences: The framework allows a robot to generate plausible sensory data for a context it hasn't experienced, based on data from another context. This is akin to a robot "imagining" what an object would feel like based on how it looks or sounds.
  • Sensor failure is not a dead end: By transferring knowledge from other modalities, robots can continue to function and recognize objects even if one sensor fails. This significantly improves robustness.
  • Cross-behavior transfer is powerful: Robots can learn about objects through actions they themselves haven't performed, by leveraging the experiences of other robots or different actions. This accelerates learning and reduces the need for exhaustive exploration.
  • Modality matters: The effectiveness of transfer is often higher when transferring within the same modality (e.g., haptic to haptic) compared to across modalities (e.g., audio to haptic). However, even cross-modal transfer can be highly effective.
  • Behavior similarity is key: Behaviors that capture similar object properties (e.g., "hold" and "lift" both involve maintaining grip and sensing force) tend to be better for knowledge transfer.
  • Clustering for efficient calibration: A method for selecting a small, representative set of objects for calibration can significantly improve the efficiency of learning transfer models, especially in time-constrained scenarios.

Key Results

  • Significant accuracy improvements: The proposed framework achieved recognition accuracies that were often comparable to, and sometimes even better than, using ground truth features from the target robot's actual interactions. For example, in one cross-perception scenario, projected haptic features from vision enabled a recognition accuracy of 42.5%, very close to the 49.6% achieved with real haptic data.
  • Cross-behavior transfer boosted performance: In cross-behavior transfer, projecting shake-haptic features to hold-haptic allowed the target robot to achieve 63.3% recognition accuracy, nearly matching the 62.5% from ground truth features.
  • Multi-source transfer is beneficial: Using multiple source modalities (e.g., audio and vision) to predict a target modality (e.g., haptic) significantly boosted recognition accuracy compared to using a single source modality.
  • Encoder-decoder methods outperform domain adaptation baselines: The proposed β-VED and β-VAE models generally outperformed a domain adaptation method (KEMA) in terms of recognition accuracy.
  • Validation on a second dataset: The framework's effectiveness was demonstrated on a different dataset without requiring any additional hyperparameter tuning, highlighting its generalizability.

Limitations and Future Work

  • Environment dependency: The current framework's learned features are tied to the robot's environment. Future work aims to develop methods for transferring knowledge that generalizes across different environments.
  • Dataset size: The datasets used were relatively small. Collecting larger datasets is suggested to better understand the data requirements for optimal performance.
  • Robot embodiment and feature representations: The current experiments primarily focused on robots with similar embodiments. Future work will explore transferring knowledge between robots with different physical forms and feature representations.
  • Scalability with multiple robots: The current β-VAE model's runtime complexity increases linearly with the number of source robots. Developing models that scale more efficiently is a future direction.
  • Incremental learning: Investigating models that can incrementally improve performance by learning from new data acquired by one of the robots is also proposed.
  • Broader task applicability: The framework was evaluated for object categorization. Future work will extend it to other tasks like object manipulation and language grounding.

Summary updated

BibTeX
@article{tatiya2020framework,
  author = {Gyan Tatiya and Ramtin Hosseini and Michael C. Hughes and Jivko Sinapov},
  title = {{A Framework for Sensorimotor Cross-Perception and Cross-Behavior Knowledge Transfer for Object Categorization}},
  journal = {Frontiers Robotics AI},
  year = {2020},
  doi = {10.3389/frobt.2020.522141},
}
Figure