Affordances in Grounded Language Learning

Stephen McGregor, KyungTae LimACL 2018

TL;DR

This paper proposes a novel approach to grounded language learning by bridging distributional semantic models (representing world knowledge) with "affordances" (perceived opportunities for action). They demonstrate a preliminary experiment mapping these two representations, suggesting a cognitively inspired path towards understanding how language acquisition might be grounded in embodied experience.

What Problem Does This Paper Address?

Current computational approaches to grounded language learning often rely on complex, opaque neural network architectures to map perceptual data (like images or object tracks) to linguistic outputs. While effective for tasks like image captioning, these methods don't offer interpretable models of how a language learner, with limited exposure, develops a rich lexicon grounded in the real world. The authors frame this as a "symbol grounding problem," where abstract symbols are somehow linked to perceptions. They propose an alternative: a "symbol ungrounding problem," where language emerges from embodied experiences and the perception of affordances, drawing inspiration from ecological psychology.

Methods

The authors propose a two-pronged approach to modeling semantic representations:

  1. Distributional Semantic Models (DSMs): They use the word2vec technique (specifically the skip-gram model) on a cleaned version of Wikipedia to learn word embeddings. These embeddings capture world knowledge by representing words as points in a high-dimensional space, where proximity and direction encode semantic relationships. This is presented as a model for the "encyclopaedic knowledge" of an experienced language user.
  2. Probabilistic Affordance Models: To simulate how a novice learner might encounter language grounded in action opportunities, they parse sentences from Wikipedia to identify words used as direct objects. For these words, they build probability distributions over associated predicates, subjects, and indirect objects. This creates a "probabilistic affordance space" where a word's representation reflects the actions it can be involved in.

They then build a rudimentary neural network to map the DSM word-vectors to these affordance probability distributions using a single dense softmax layer. This network learns to approximate the affordance distribution for a given word-vector.

Datasets

  • Wikipedia: A cleaned-up version of Wikipedia (approximately 9.08 million word types, 1.87 billion word tokens, 87.2 million sentences) was used for both training the distributional semantic word-vectors and for extracting affordance information.
  • Rosch (1975) conceptual categories: Five prototypical words from five conceptual categories (VEHICLES, CLOTHING, TOOLS, FURNITURE, FRUIT) were selected for the preliminary experiment.

Key Insights and Ideas

  • Affordances as a bridge: The core idea is that affordances—the perceived opportunities for action—can serve as a crucial link between abstract linguistic symbols and embodied, environmental experiences.
  • "Symbol Ungrounding" vs. "Symbol Grounding": The paper advocates for a shift in perspective from how abstract symbols are grounded to how they emerge from ungrounded, embodied interactions.
  • Two complementary representations: The work highlights the value of combining broad world knowledge (from DSMs) with action-oriented, situated knowledge (from affordances) for a more complete model of language learning.
  • Syntactic cues as a proxy: While acknowledging the ultimate goal of more situated methods, the paper uses syntactic relationships (predicate, subject, indirect object) as a computational proxy for simulating how an agent might encounter words in relation to objects and actions.

Key Results

  • Clustering: Both word-vectors (clustered using k-means) and affordance representations (clustered using k-medoids) showed some alignment with human-judged categories, though not perfectly. For instance, furniture and tools sometimes clustered together, and fruits were grouped with some furniture and tools, orbiting the word "orange."
  • Mapping Accuracy: The preliminary neural network mapping word-vectors to affordance matrices achieved moderate accuracy. When mapping a word-vector to the same word's affordance matrix, accuracy was around 0.12 (total) and 0.64 (across classes). When mapping to another word of the same class, accuracy was higher (0.2 for total, 0.8 for VEHICLES, 1.0 for FRUIT). This suggests that while exact word-to-word mapping is challenging, the model can capture some categorical information.

Limitations and Future Work

  • Limited Scope: The experiments used a small vocabulary and a limited set of grammatical classes, contrasting with more ambitious multi-modal tasks.
  • Computational Proxy: The use of syntactic information is acknowledged as a stand-in for more direct, situated interactions.
  • Future Directions:
    • Parameter Exploration: Experimenting with different word2vec parameters, affordance grammatical classes, target vocabulary, and network architectures.
  • Situated Methodology: Developing more direct, situated methods for modeling affordances, potentially incorporating human-robot interaction, gaze, and gesture.
  • Robotic Applications: Exploring how "ontogenetic ritualisation" plays a role in language learning.
  • Mapping to Perception: Investigating the use of affordance-based models for mapping between low-level perceptual input (like images) and abstract linguistic representations, potentially as a component of autoencoder networks. The authors express interest in mapping from images to affordances.

Summary updated

BibTeX
@inproceedings{mcgregor2018affordances,
  author = {Stephen McGregor and KyungTae Lim},
  title = {{Affordances in Grounded Language Learning}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2018},
}
Figure