TL;DR
This paper introduces Feature2Vec, a novel method that bridges the gap between opaque, high-dimensional word embeddings from text and interpretable, human-generated property knowledge. By mapping human-elicited features into a pre-trained semantic space, Feature2Vec allows for richer concept representations and offers a promising avenue for extending existing property norm datasets.
What Problem Does This Paper Address?
Modern Natural Language Processing (NLP) heavily relies on distributional semantic models, like word embeddings (e.g., GloVe, word2vec), which learn word meanings from vast amounts of text. These models represent words as dense vectors in a high-dimensional space, where proximity indicates semantic similarity. However, a significant drawback is their "opaqueness" – the individual dimensions of these vectors lack clear, human-understandable meaning, making them difficult to interpret and connect to cognitive processes.
In contrast, human conceptual knowledge is often captured through property norming studies. These studies ask people to list properties of concepts (e.g., "a robin has wings," "a robin is red"). While these norms are highly interpretable and cognitively plausible, they are labor-intensive to collect, resulting in limited datasets in terms of the number of concepts and the completeness of their property listings. This paper aims to combine the strengths of both approaches: the rich coverage of distributional models and the interpretability of human property knowledge.
Methods
The core contribution is Feature2Vec, a computational framework that adapts the skip-gram architecture (used in word2vec) to model human property knowledge. Instead of predicting context words from a target word, Feature2Vec predicts properties from a concept.
Here's how it works:
- Pre-trained Word Embeddings: The model starts with pre-trained GloVe embeddings for words. These embeddings are fixed during Feature2Vec training, meaning their weights are not updated. This anchors the learned property representations in a well-established semantic space.
- Property Embeddings: For each property (e.g., "has wings"), a trainable embedding vector is created. These vectors are randomly initialized and are the primary focus of the learning process.
- Skip-Gram Adaptation: The standard skip-gram architecture is modified. The "target word" is now a concept, and the "context words" are its associated properties. The network learns to predict properties given a concept.
- Negative Sampling: Similar to word2vec, negative sampling is used to make training efficient. Instead of sampling negative context words, negative properties are sampled.
- Loss Function: The training uses a binary cross-entropy loss, aiming to maximize the probability of correct property-concept associations and minimize the probability of incorrect ones.
The key innovation is mapping property-based information into the existing distributional semantic space, rather than trying to predict properties from it (as in previous regression-based methods).
Datasets
The paper uses two established property norm datasets for training and evaluation:
- McRae property norms: Contains 541 concepts and 2526 features.
- CSLB norms: Contains 638 concepts and 2725 features.
For both datasets, a feature is included if it was elicited by at least five participants. The "production frequency" of a feature for a concept (how many participants listed it) is used as a measure of its relevance.
The paper also uses pre-trained GloVe embeddings trained on the Common Crawl corpus, providing a large vocabulary (685,000 tokens) with 300-dimensional vectors.
Key Insights and Ideas
- Bridging the Semantic Gap: Feature2Vec successfully integrates human-centric, interpretable property knowledge with the data-driven, opaque representations of distributional semantics. This is crucial for making NLP models more cognitively plausible.
- Interpretable Embeddings for Properties: The learned property embeddings are aligned with the GloVe space, meaning that semantically similar properties will have similar embeddings. This allows for direct comparison and analysis of feature relationships.
- Compositionality of Concepts: By averaging the learned feature embeddings for a concept, the model can approximate the concept's original GloVe embedding. This suggests that human-defined properties can indeed contribute to building meaningful concept representations.
- Assisting Property Norming: The model's ability to predict plausible features, even for concepts not in the original training data, highlights its potential as a tool to guide and extend expensive human property norming efforts.
Key Results
- Predicting Feature Vectors: While not a direct comparison due to methodological differences, the paper shows that concept representations constructed by averaging Feature2Vec-learned feature embeddings are reasonably well-represented among nearest neighbors in the GloVe space.
- Predicting Property Knowledge: Feature2Vec significantly outperforms a baseline Partial Least Squares Regression (PLSR) method on predicting property knowledge. For instance, on the McRae dataset, Feature2Vec achieves 45.39% Top 10 prediction accuracy compared to PLSR 50's 29.79% and PLSR 120's 43.26%. Similar improvements are seen on the CSLB dataset.
- Plausible Feature Predictions: The qualitative analysis shows that Feature2Vec can generate highly plausible features for concepts, even for words not present in the original property norm datasets. These predicted features often align with common sense, demonstrating the model's ability to generalize.
Limitations and Future Work
- Semi-Complete Norms: The authors acknowledge that property norm datasets are inherently incomplete. Feature2Vec's predictions can help identify missing properties, but the ground truth remains limited.
- Iterative Refinement: A key future direction is to use Feature2Vec in an iterative, semi-supervised manner. Predicted features with high confidence could be verified by humans and added to the norms, which are then used to retrain Feature2Vec, creating a virtuous cycle of dataset expansion and model improvement.
- Extending Vocabulary: The model's ability to generate plausible features for out-of-norm words suggests it can be used to build interpretable feature-based information for new words in a practical and efficient way.
Summary updated