Robot Concept Acquisition Based on Interaction Between Probabilistic and Deep Generative Models

Ryo Kuniyasu, Tomoaki Nakamura, Tadahiro Taniguchi, Takayuki NagaiFrontiers of Computer Science 2021 · DOI

TL;DR

This paper introduces a novel method for robots to acquire multimodal concepts by integrating a probabilistic model (MLDA) with a deep generative model (Multinomial VAE). This integration allows for unsupervised clustering, feature learning, and cross-modal inference, enabling robots to learn from ambiguous sensory data without pre-trained feature extractors.

What Problem Does This Paper Address?

Robots need to understand the world by forming concepts from various sensory inputs (like images and language). While existing methods like Multimodal Latent Dirichlet Allocation (MLDA) are good at clustering and cross-modal inference, they often rely on pre-designed or pre-trained feature extractors. This means the robot isn't learning the best way to represent the sensory data for concept formation in its specific environment. Furthermore, previous approaches couldn't predict sensory data from other modalities (e.g., generate an image from text). This paper addresses the need for a truly unsupervised approach where robots can simultaneously learn to extract relevant features and form concepts from multimodal data, enabling them to adapt to their environment and perform cross-modal inference.

Methods

The core of this work is the integration of two key components:

  • Multimodal Latent Dirichlet Allocation (MLDA): A probabilistic graphical model used for concept formation, clustering, and cross-modal inference. It assumes observations follow multinomial distributions.
  • Multinomial Variational Autoencoder (MNVAE): An extension of the standard Variational Autoencoder (VAE). The key innovation here is that the latent variables of the MNVAE follow a multinomial distribution, making them compatible with the MLDA's assumptions. This allows the MNVAE to learn feature representations from sensory data (like images) that can be directly used by the MLDA.

The MNVAE acts as a feature extractor, compressing raw sensory data into latent representations. The MLDA then uses these representations, along with other modalities (like text), to infer categories (concepts). Crucially, the two models interact: the MNVAE provides features to the MLDA, and the MLDA provides feedback to the MNVAE, guiding it to learn features that are more suitable for clustering and concept formation. This interaction is achieved through parameter exchange, optimizing both models simultaneously in an unsupervised manner.

Datasets

The experiments used a dataset previously introduced in (Aoki et al., 2016). This dataset comprises images, tactile sensor values, and sound data from 499 objects across 81 categories. For this study, only the image and utterance data were utilized. Images were captured by an RGB camera and preprocessed (object detection, resizing to 144x120). Utterances were processed using a phoneme recognizer and then segmented into words using unsupervised methods, forming bag-of-words (BoWs) representations.

Key Insights and Ideas

  • Unsupervised Feature Learning for Concept Formation: The most significant insight is that by integrating a deep generative model (MNVAE) with a probabilistic concept formation model (MLDA), robots can learn to extract features that are directly useful for concept formation, without needing pre-trained models or manual feature engineering.
  • Bridging Probabilistic and Deep Learning: The paper successfully bridges the gap between probabilistic models (MLDA) and deep generative models (VAE) by introducing the Multinomial VAE, enabling their seamless integration. This allows for the interpretability of MLDA with the representational power of deep learning.
  • Cross-Modal Inference from Words to Images: The integrated model demonstrates the ability to generate images from textual descriptions, showcasing effective cross-modal inference. This is a crucial step towards robots understanding and interacting with the world based on language.
  • Improved Concept Representation: The interaction between MNVAE and MLDA leads to a more structured latent space where data points belonging to the same category are clustered more tightly, indicating better concept representation.
  • Adaptability to Ambiguous Data: The method is shown to be effective even with noisy data (e.g., speech recognition errors) and limited training data, highlighting its robustness and practicality for real-world robotic applications.

Key Results

  • Classification Accuracy: The proposed MNVAE + MLDA model achieved a classification accuracy of 0.660 ± 0.018, which is comparable to the supervised MLDA baseline (0.688 ± 0.014) and significantly better than the JMVAE + GMM baseline (0.465 ± 0.013).
  • Improved Latent Space Structure: Visualizations (Figure 5) show that the latent space generated by MNVAE + MLDA is more organized, with better separation of categories compared to using MNVAE alone.
  • Effective Cross-Modal Generation: The model successfully generated images from words, and a subjective evaluation (Table 3) indicated that these generated images were often preferred as more representative of the given word compared to those generated by the JMVAE.
  • Iterative Improvement: Classification accuracy improved with iterative parameter exchange between the MNVAE and MLDA, demonstrating the benefit of their interaction.
  • Image Reconstruction: While Structural Similarity Index Measure (SSIM) for image reconstruction decreased with interaction, the reconstructed images were still distinguishable, suggesting that the latent variables captured both reconstruction information and category-specific features.

Limitations and Future Work

  • Dataset Size: The authors acknowledge that the dataset size (499 objects) was not sufficiently large, which may have impacted the quality of generated images.
  • Category Bias: The dataset exhibited a bias in the number of objects per category, which could lead to misclassifications.
  • Future Work:
    • Incorporating More Modalities: Extending the model to include tactile and auditory information alongside images and words.
  • Language Model Integration: Developing a model that can jointly learn language models (speech recognizers) and concept formation.
  • Online and Interactive Learning: Implementing online learning and interactive learning mechanisms to address biased environments and improve adaptation.
  • Exploring Cognitive Science Concepts: Investigating how the MLDA-based approach can replicate more complex cognitive phenomena related to concepts.

Summary updated

BibTeX
@article{kuniyasu2021robot,
  author = {Ryo Kuniyasu and Tomoaki Nakamura and Tadahiro Taniguchi and Takayuki Nagai},
  title = {{Robot Concept Acquisition Based on Interaction Between Probabilistic and Deep Generative Models}},
  journal = {Frontiers of Computer Science},
  year = {2021},
  doi = {10.3389/fcomp.2021.618069},
}
Figure