TL;DR
This paper tackles the challenge of identifying "semantically compatible" words – words that can plausibly refer to the same thing in a normal context (e.g., "cat" and "hindrance" are compatible, but "cat" and "dog" are not). They introduce a new dataset of human compatibility ratings and a neural network model that learns to predict compatibility, outperforming standard word embedding similarity measures.
What Problem Does This Paper Address?
Current computational models of word meaning, like those based on distributional embeddings (e.g., Word2Vec), are excellent at capturing semantic relatedness. However, relatedness is a broad concept. For instance, "dog," "puppy," and "animal" are all related to "dog," but they imply different things. Knowing "Fido is a dog" allows us to infer he's an "animal" and not a "cat," but whether he's a "puppy" is uncertain. This paper argues that a more nuanced understanding of semantic compatibility is crucial for many NLP tasks. Compatibility, defined as the ability for two words to refer to the same thing in a "reasonably normal state of affairs," goes beyond simple relatedness. For example, "husband" and "hindrance" are compatible, despite not being semantically close. Existing models struggle to distinguish between relatedness and compatibility, leading to potential errors in tasks like coreference resolution, entailment detection, and even understanding negation.
Methods
The authors propose a supervised neural network approach. Their core idea is to take pre-trained distributional word embeddings (from Word2Vec) as input and learn a new embedding space specifically optimized for compatibility detection. They explore several model architectures, including:
- Direct Models (1L direct, 2L direct): These models directly combine mapped (or original) word embeddings.
- Interaction Models (1L interaction, 2L interaction): These models explicitly model the pairwise interactions between features derived from the input embeddings. This is a key innovation, allowing the model to learn how features from two words combine to determine compatibility.
- Interaction Direct Models (1L int. direct, 2L int. direct): These combine both mapped representations and their interaction vector.
- Mono Models (1L mono, 2L mono): These serve as baselines, considering only one word in a pair to understand if compatibility can be predicted from individual word properties alone.
The "2L interaction" model, which uses a shared mapping function for both words and then constructs a vector of pairwise interactions, emerges as the most promising.
Datasets
The paper introduces a new, publicly available dataset of human compatibility ratings.
- Benchmark Construction: They started with a list of 299 words covering various categories (animals, professions, tools, etc.). Random pairs were generated, and then about 18,000 pairs were annotated by human raters on the CrowdFlower platform.
- Annotation Process: Initially, a pilot study used a 7-point scale. Due to high correlation with binary judgments, they adopted the 7-point scale for the main survey. Raters were asked to consider "real-life scenarios" and choose the most common sense if ambiguity arose.
- Dataset Size: The final dataset contains 17,973 pairs, each rated by 10 participants, with a median compatibility score of 1.10 (indicating a strong bias towards incompatibility in random pairs).
Key Insights and Ideas
- Compatibility is Distinct from Relatedness: The paper convincingly demonstrates that semantic compatibility is a unique property of word meaning, not fully captured by standard semantic relatedness measures.
- Interaction is Key: The success of models that explicitly model the interaction between word features (like the 2L interaction model) highlights that compatibility often arises from how two concepts relate to each other, not just their individual properties.
- Category Information Matters: Visualizations show that the learned features in the 2L interaction model group words by general semantic categories (e.g., "animal," "human"), suggesting that high-level category information is crucial for predicting compatibility.
- Co-hyponymy is a Partial Indicator: While co-hyponyms (words under the same hypernym, like "dog" and "cat") are often incompatible, this is not a perfect rule. The paper shows that a simple co-hyponymy classifier performs poorly, but a strategy of treating co-hyponyms as compatible and non-cohyponyms as incompatible performs better, though still not sufficient.
Key Results
- Superior Performance: The proposed supervised models, particularly the "2L interaction" model, significantly outperform direct cosine similarity of word embeddings on compatibility detection.
- High Scores: The 2L interaction model achieves an F1 score of 87 for incompatibility detection and 66 for compatibility detection, with a Pearson correlation of 0.72 with human ratings.
- Beyond Individual Words: The supervised models outperform "mono" models that only consider one word in a pair, confirming that compatibility is a relational property.
- Data Skew: The dataset is heavily skewed towards incompatibility, with a median score of 1.10 on a 7-point scale.
Limitations and Future Work
- "Reasonably Normal State of Affairs": The authors acknowledge that their definition of compatibility relies on a vague notion of "reasonably normal state of affairs," which they use to exclude science-fiction scenarios.
- Data Distribution: The generated pairs might not perfectly reflect the natural distribution of compatibility decisions encountered in everyday life.
- Future Work: The authors suggest that computational models of compatibility can significantly improve various NLP tasks, including coreference resolution, question answering, modeling plausibility, and negation. They plan to explore the contributions of compatibility in these areas.
Summary updated