Abstract
The compositionality degree of multiword expressions indicates to what extent the meaning of a phrase can be derived from the meaning of its constituents and their grammatical relations. Prediction of (non)-compositionality is a task that has been frequently addressed with distributional semantic models. We introduce a novel technique to blend hierarchical information with distributional information for predicting compositionality. In particular, we use hypernymy information of the multiword and its constituents encoded in the form of the recently introduced Poincaré embeddings in addition to the distributional information to detect compositionality for noun phrases. Using a weighted average of the distributional similarity and a Poincaré similarity function, we obtain consistent and substantial, statistically significant improvement across three gold standard datasets over state-of-the-art models based on distributional information only. Unlike traditional approaches that solely use an unsupervised setting, we have also framed the problem as a supervised task, obtaining comparable improvements. Further, we publicly release our Poincaré embeddings, which are trained on the output of handcrafted lexical-syntactic patterns on a large corpus.
Introduction
In Natural Language Processing (NLP), the principle of compositionality—the idea that the meaning of a complex expression is determined by the meanings of its constituent parts—is a fundamental assumption. While this holds for literal phrases like "data format," it fails for idiomatic expressions such as "couch potato" or "rat race." Distinguishing between literal and figurative meanings is crucial for downstream tasks like machine translation and information retrieval.
Traditional approaches rely on distributional semantics, which assumes that words in similar contexts share meanings. However, context alone often fails to capture the structural nuances of idiomaticity. This paper, presented at NAACL-HLT 2019, proposes a hybrid approach that integrates hierarchical "is-a" (hypernymy) relationships into distributional models to improve compositionality prediction.
Methodology: Blending Distributional and Hierarchical Information
The researchers proposed a hybrid model, MODEL-DP, that combines two distinct types of semantic information:
- The Distributional Component: Standard word embeddings (e.g., Word2Vec) measure the similarity between the vector of a phrase and the combined vectors of its constituents.
- The Hypernymy Component: The authors hypothesized that for literal phrases, the "category" (hypernym) of the whole phrase should align with the hypernym of its head word. For instance, "art school" and "school" both fall under "educational institution," whereas "hot dog" (food) and "dog" (animal) do not.
To represent these hierarchies, the authors utilized Poincaré embeddings in hyperbolic space. Hyperbolic space is uniquely suited for representing tree-like structures, allowing for high-precision taxonomic mapping in low dimensions. The final compositionality score is a weighted average (controlled by parameter $\alpha$) of distributional and Poincaré similarities. Additionally, the authors applied noise filtering by removing the top $m\%$ of the most frequent (and often overly abstract) "is-a" pairs.
Experimental Setup
The model was evaluated on three gold-standard English datasets where humans manually rated phrases for compositionality:
- Reddy (RD): 90 compounds (e.g., "silver medal").
- Reddy++ (RD++): An extension of RD with 180 total compounds.
- Farahmand (FD): A larger set of 1,042 compounds extracted from Wikipedia.
Key Results and Insights
The hybrid model consistently outperformed models using distributional data alone, showing an absolute improvement of 2-4% in Spearman’s rank correlation across all datasets.
Key insights from the study include:
- The Hypernymy Hypothesis: Taxonomic consistency is a strong signal for compositionality. If the "is-a" branch of the phrase deviates sharply from its components, it is likely an idiom.
- Hyperbolic Efficiency: Poincaré embeddings effectively capture complex hierarchies in just 50 dimensions while maintaining high precision.
- Noise Reduction: Filtering out the most frequent, abstract hypernyms (e.g., "thing") improves model robustness, particularly on larger datasets like Farahmand (FD).
- Supervised Success: When framed as a supervised task using regression, the inclusion of Poincaré features led to even higher accuracy, beating previous supervised benchmarks by a considerable margin.
Conclusion
By blending distributional semantics with hyperbolic hierarchical embeddings, this work achieves state-of-the-art results in compositionality prediction. While challenges remain—such as handling "semi-idioms" like "hard disk" or dealing with data sparsity for rare phrases—the integration of structural taxonomic information represents a significant step forward. The authors have publicly released their pre-trained Poincaré embeddings to support further research in this area.
Summary updated