TL;DR
This paper introduces a novel method for extracting "affordances" – what actions are possible and sensible in a given situation – by leveraging word embeddings trained on Wikipedia. By treating these embeddings as a common-sense knowledge base, the authors demonstrate that autonomous agents can significantly reduce their action search space, leading to faster learning and more human-like decision-making, particularly in text-based environments.
What Problem Does This Paper Address?
Autonomous agents, especially those operating in complex environments like text-based adventure games, often face an overwhelming number of possible actions. This leads to inefficient exploration and slow learning as agents try out many nonsensical or impossible actions. Humans, on the other hand, possess a rich "common sense" that allows them to quickly infer what actions are feasible and appropriate in a given context. The challenge is to imbue artificial agents with this common-sense understanding of affordances to make their learning and decision-making more efficient and human-like.
Methods
The core of the paper's contribution lies in using word embeddings (specifically, word2vec trained on Wikipedia) as a proxy for common-sense knowledge. The authors propose two main ways to leverage these embeddings for affordance extraction:
- Verb/Noun Affordances: They define an "affordance vector" by averaging the difference between vectors of canonical verb-noun pairs (e.g.,
vector('sing') - vector('song')). This vector captures the semantic relationship of an action being appropriate for an object. By adding this affordance vector to an object's word vector, they can query the embedding space to find verbs that are semantically "affordant" to that object. - Graspable Objects (Manipulability): To determine if an object is physically manipulable, they define axes in the embedding space using pairs of words that represent contrasting concepts (e.g.,
vector('forest') - vector('tree')). By projecting object word vectors onto these axes, they can estimate an object's manipulability, distinguishing things like "pencil" from "shadow."
These affordance-based methods are then integrated into a Q-learning agent to reduce its action and object selection spaces.
Datasets
- Training Data: A Wikipedia corpus, preprocessed with part-of-speech tagging.
- Evaluation Data: 50 text-based adventure games compatible with the Infocom Z-machine.
Key Insights and Ideas
- Wikipedia as a Common Sense Knowledge Base: The paper argues that the vast, unstructured data in Wikipedia, when processed into word embeddings, implicitly encodes a rich amount of common-sense knowledge that can be mathematically queried.
- Mathematical Manipulation of Semantics: The idea that word vectors can be manipulated using linear algebra to extract meaningful semantic relationships (like "what can you do with X?") is a powerful concept.
- Affordances for Action Space Reduction: By identifying sensible actions, agents can drastically prune their search space, leading to more efficient learning. This is crucial for domains with large action sets.
- Distinguishing "Reasonable" from "Possible": The method goes beyond just identifying physically possible actions to identifying those that are "reasonable" or make sense in context, a key aspect of human intelligence.
- Implicit Knowledge in Embeddings: The authors note that word embeddings can capture relationships that even human authors might not consciously articulate, offering a unique advantage.
Key Results
- Improved Learning Speed: The affordance-based agent significantly reduced the total number of steps needed to learn compared to a brute-force baseline.
- Enhanced Performance: The agent using affordance extraction achieved higher peak performance (up to 57.44% of possible game score) compared to the baseline agent (20.84%).
- Broader Game Success: The affordance-based agent scored points on 18 out of 49 games, while the baseline scored on 12 out of 49.
- Human-like Action Selection: The agent's action selections began to resemble those a human would choose.
- Comparison to Other Methods: Affordance-based reduction outperformed other tested methods like intrinsic rewards, random reduction, ConceptNet reduction, and co-occurrence reduction in the evaluated games.
Limitations and Future Work
- Polysomy: The current method is not robust to words with multiple meanings (polysomy), leading to unpredictable behavior (e.g., "apple" referring to the company instead of the fruit).
- Prepositional Commands: The agent cannot construct complex commands involving prepositions (e.g., "put book on shelf").
- Implicit Goal Inference: The agent struggles with games requiring the inference of proper terms for manipulable objects or complex sequences of events to achieve goals.
- Future Directions: The authors suggest that these principles are applicable beyond text-based environments and could be extended to systems like self-driving cars, domestic robots, and UAVs, leveraging advances in object recognition and semantic segmentation. They also envision further research into "one-shot learning" for text-based games.
Summary updated