Firearms and Tigers are Dangerous, Kitchen Knives and Zebras are Not: Testing whether Word Embeddings Can Tell

Pia Sommerauer, Antske FokkensEMNLP 2018

TL;DR

This paper introduces a novel method to probe what kind of semantic information is encoded in word embeddings, moving beyond traditional analogy tasks. By comparing supervised classifiers to full-vector similarity, they found that embeddings capture functional and interaction-based properties better than perceptual ones like color. This work offers a more nuanced understanding of word embedding capabilities, crucial for interpreting their success in downstream NLP tasks.

What Problem Does This Paper Address?

Word embeddings, like Word2Vec, are foundational to modern Natural Language Processing (NLP), significantly boosting performance across various tasks. However, despite their widespread use and impressive results on tasks like semantic similarity and analogy, our understanding of what specific semantic information these embeddings actually capture remains limited. Traditional evaluation methods often rely on full-vector comparisons or analogy tasks, which have been criticized for not necessarily reflecting true semantic understanding. For instance, high performance on analogy tasks doesn't guarantee that embeddings capture specific semantic properties. This paper aims to bridge this gap by developing a method to directly test which semantic properties are encoded in word embeddings, and how.

Methods

The core of the paper's contribution is a new methodology for evaluating word embeddings. They propose a two-pronged approach:

  1. Supervised Classification: They train binary classifiers (logistic regression and a simple neural network) to distinguish between words that possess a specific semantic property (positive examples) and those that do not (negative examples). The input to these classifiers are the word embedding vectors.
  2. Full-Vector Similarity: They compare the classifier performance to a baseline approach that relies on the cosine similarity between word vectors. Specifically, they calculate the centroid of positive examples and measure how close other words are to this centroid.

The key idea is that if a classifier can identify a property while full-vector similarity struggles, it suggests that the property is encoded in more specific dimensions or patterns within the embedding, rather than just general semantic relatedness. Conversely, if both methods fail, the property is likely not well-represented.

To facilitate this, they extend an existing human-elicited semantic property dataset (CSLB) by adding carefully verified negative examples, often sourced through crowdsourcing. This extension is crucial for robust evaluation, moving beyond simple logical exclusion of properties.

Datasets

  • Extended CSLB Data: The primary dataset used is the Centre for Speech, Language and the Brain (CSLB) concept property norms dataset. This dataset was extended with verified negative examples, both through logical exclusion and crowdsourcing. The authors specifically mention creating additional sets for words exhibiting food-animal polysemy to test nuanced understanding.
  • Pretrained Word2vec Model: The experiments utilize a pretrained Word2vec model trained on the Google News corpus, which has 300 dimensions.

Key Insights and Ideas

  • Beyond General Similarity: The paper strongly suggests that word embeddings can encode information beyond simple semantic similarity. Classifiers can sometimes identify properties that are not easily captured by just looking at the overall vector space.
  • Functional vs. Perceptual Properties: Embeddings seem to better capture properties related to how entities interact with the world (e.g., "is dangerous," "is used in cooking," "has wheels") than perceptual properties like color ("is black," "is red").
  • The Role of Context and Dimensions: Neural networks, with their ability to learn from patterns across dimensions, show promise in identifying properties that might not be linearly separable or obvious from individual dimensions alone.
  • Bias in Datasets: The authors acknowledge that biases in negative example generation (e.g., many negative examples for "is_black" being animals) can lead classifiers to learn superficial distinctions rather than true semantic properties. This highlights the importance of carefully curated datasets.
  • Polysemy is Tricky: Distinguishing between different senses of a word (e.g., "chicken" as an animal vs. food) proved challenging, with classifiers sometimes only identifying the dominant sense rather than both.

Key Results

  • Color Properties Perform Poorly: Properties like "is_yellow" and "is_red" consistently yielded low F1-scores across all approaches, with classifiers showing only a slight improvement over nearest neighbors.
  • Functional Properties Perform Well: Properties like "does_kill," "is_dangerous," and "is_used_in_cooking" showed significantly higher F1-scores for classifiers compared to nearest neighbors, indicating better encoding. For "is_dangerous," the difference was substantial (60-19 points).
  • "Has_wheels" and "is_found_in_seas" Show Nuance: These properties, while having high nearest-neighbor performance due to taxonomic correlations, also showed good classifier performance, suggesting that classifiers can make finer-grained distinctions (e.g., identifying specific types of vehicles or distinguishing seawater from freshwater fish).
  • "Is_black" and "made_of_wood" are Anomalies: These visual properties showed surprisingly high classifier performance, attributed to potential biases in the negative examples (e.g., many animals being "is_black" or food/animals being "made_of_wood").
  • Crowd-Only Datasets Reduce Bias: When using smaller, crowd-only datasets that were less biased, the performance of classifiers dropped marginally, while nearest neighbors' performance rose, suggesting the original biases were indeed influencing the results.

Limitations and Future Work

  • Dataset Size and Balance: The primary limitation acknowledged is the limited size and potential imbalance of the semantic datasets, which restricts the ability to draw broad conclusions and necessitates leave-one-out cross-validation.
  • Parameter Tuning: No parameter tuning was performed, which could have further optimized classifier performance.
  • Scope of Properties: The study focused on a small subset of semantic properties, limiting generalizability.

Future work includes:

  • Expanding Datasets: Incorporating other relevant datasets and refining existing ones with graded judgments (e.g., "mostly," "possibly").
  • Robust Evaluation: Conducting experiments with different train/test splits to overcome leave-one-out limitations.
  • Parameter Optimization: Performing careful parameter tuning for classifiers.
  • Investigating Context: Exploring how information encoded in linguistic context relates to what's present in word embeddings.
  • Exploring Other Methods: Applying methods like those from Faruqui et al. (2015) for deeper insights into embedding dimensions.

Summary updated

BibTeX
@inproceedings{sommerauer2018firearms,
  author = {Pia Sommerauer and Antske Fokkens},
  title = {{Firearms and Tigers are Dangerous, Kitchen Knives and Zebras are Not: Testing whether Word Embeddings Can Tell}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
}
Figure