On the Compositionality Prediction of Noun Phrases using Poincaré Embeddings

Abhik Jana, Dmitry Puzyrev, Alexander Panchenko, Pawan Goyal, Chris Biemann, Animesh Mukherjee

Introduction

The principle of compositionality suggests that the meaning of a complex linguistic expression is a function of the meanings of its constituent parts and the rules used to combine them. While this holds for literal phrases like "green tree," it fails for idiomatic expressions such as "couch potato" or "rat race," where the overall meaning is non-compositional. Accurately predicting the degree of compositionality is vital for downstream Natural Language Processing (NLP) tasks, including machine translation and word sense disambiguation.

Traditional distributional models, such as Word2Vec or GloVe, typically assess compositionality by comparing the vector representation of a phrase to the vectors of its individual components. However, these models often lack the nuanced semantic information required to distinguish between literal and figurative uses. This paper proposes a novel hybrid approach that augments standard distributional semantics with hierarchical taxonomic information (hypernymy) represented in hyperbolic space using Poincaré embeddings.

Methodology

The proposed model, referred to as MODEL-DP, utilizes a hybrid scoring metric that integrates two distinct semantic signals:

  1. Distributional Component: This follows the standard approach of calculating the cosine similarity between the vector of the entire phrase and the sum of the vectors of its constituent words.
  2. Hypernymy Component: The authors hypothesize that literal phrases share common hypernyms with their "head" words. For instance, "art school" and "school" both fall under the hypernym "educational institution." In contrast, idioms like "hot dog" share few, if any, specific hypernyms with "dog."

To implement the hypernymy component, the authors extracted 27.6 million hyponym-hypernym pairs from a large corpus using Hearst patterns. These pairs were used to train Poincaré embeddings. Unlike Euclidean space, hyperbolic space (specifically the Poincaré ball model) is mathematically suited for representing hierarchical, tree-like structures with minimal distortion because its volume grows exponentially toward the boundary. The final compositionality score is a weighted average of the distributional and Poincaré similarity scores, controlled by a parameter $\alpha$.

Evaluation

The models were evaluated on three benchmark datasets:

  • Reddy (RD): 90 compounds with human ratings from 0 to 5.
  • Reddy++ (RD++): An extension of the Reddy dataset containing 180 adjective-noun pairs.
  • Farahmand (FD): A larger dataset of 1,042 compounds with expert binary judgments.

Performance was measured using Spearman’s rank correlation ($\rho$) between the model's predicted scores and human gold-standard judgments.

Results and Discussion

Initial experiments with base models revealed that while distributional models provide a solid foundation, Poincaré embeddings alone are insufficient for compositionality prediction.

Performance of various base models across datasets. Note the low performance of Poincaré embeddings when used in isolation.

As shown in Figure 1, W2V-CBOW emerged as the strongest distributional baseline. However, the integration of hierarchical information in MODEL-DP provided a significant performance boost. The authors found that a weighting of $\alpha = 0.4$ (placing more weight on the distributional component but significantly incorporating the hierarchical signal) yielded the best results.

Comparison of the best baseline (W2V-CBOW) against the proposed hybrid model (MODEL-DP) on reduced and full datasets.

Figure 5 illustrates the consistent superiority of MODEL-DP over the W2V-CBOW baseline across both reduced and full dataset versions. The hybrid approach achieved a 2-4% absolute improvement in Spearman’s correlation. This suggests that "context" (from distributional data) and "hierarchy" (from Poincaré embeddings) provide complementary semantic information.

Conclusion and Future Work

This research demonstrates that incorporating taxonomic hierarchies via hyperbolic embeddings significantly enhances the prediction of noun phrase compositionality. By leveraging the natural fit of Poincaré balls for hierarchical data, the model better identifies literal phrases through hypernym consistency.

Despite these gains, challenges remain. The model's performance is limited by the sparsity of hypernyms in the training corpus—if Hearst patterns fail to identify a phrase's hypernyms, the model reverts to standard distributional methods. Additionally, "semi-idiomatic" phrases like "hard disk," which share hypernyms with their components despite having specialized meanings, remain difficult to classify. Future work will explore extending this approach to other languages and grammatical structures beyond noun phrases.

Summary updated

Abstract

The compositionality degree of multiword expressions indicates to what extent the meaning of a phrase can be derived from the meaning of its constituents and their grammatical relations. Prediction of (non)-compositionality is a task that has been frequently addressed with distributional semantic models. We introduce a novel technique to blend hierarchical information with distributional information for predicting compositionality. In particular, we use hypernymy information of the multiword and its constituents encoded in the form of the recently introduced Poincaré embeddings in addition to the distributional information to detect compositionality for noun phrases. Using a weighted average of the distributional similarity and a Poincaré similarity function, we obtain consistent and substantial, statistically significant improvement across three gold standard datasets over state-of-the-art models based on distributional information only. Unlike traditional approaches that solely use an unsupervised setting, we have also framed the problem as a supervised task, obtaining comparable improvements. Further, we publicly release our Poincaré embeddings, which are trained on the output of handcrafted lexical-syntactic patterns on a large corpus.
BibTeX
@inproceedings{jana2019compositionality,
  author = {Abhik Jana and Dmitry Puzyrev and Alexander Panchenko and Pawan Goyal and Chris Biemann and Animesh Mukherjee},
  title = {{On the Compositionality Prediction of Noun Phrases using Poincaré Embeddings}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.18653/v1/p19-1316},
}
Figure