Composing Noun Phrase Vector Representations

Aikaterini-Lida Kalouli, Valeria de Paiva, Richard CrouchACL 2019

TL;DR

This paper introduces two novel constraints for composing noun phrase (NP) vector representations: prioritizing semantic contribution over syntactic roles and selecting relevant dimensions from modifier embeddings. These constraints significantly improve the expressiveness of NP representations, outperforming 11 other methods across 6 evaluation tasks.

What Problem Does This Paper Address?

Representing the meaning of phrases and sentences using word embeddings is a long-standing challenge in Natural Language Processing (NLP). While word embeddings like Word2Vec and GloVe have become standard, effectively composing them to capture phrase meaning remains an open problem. Existing approaches fall into two categories: non-compositional (treating phrases as atomic units) and compositional (combining word embeddings). Non-compositional methods are not scalable, while compositional methods often struggle to accurately reflect the nuanced meaning of phrases, sometimes failing to capture semantic contributions beyond simple syntactic roles. This paper aims to bridge this gap by proposing linguistically informed constraints that improve compositional phrase representations.

Methods

The authors propose two novel constraints for composing noun phrase vectors:

  • Constraint One: Semantic Contribution: This constraint shifts focus from syntactic roles (head/modifier) to semantic roles. It posits that the word with the greater semantic contribution to the phrase's meaning should be treated as the "semantic head," regardless of its syntactic role. This is determined by comparing the similarity of phrase components to a target synonym.
  • Constraint Two: Dimensions' Selection: This constraint refines the composition process by selecting only the dimensions of the semantic modifier's embedding that are most relevant to the semantic head. This is achieved by comparing corresponding dimensions of the head and modifier embeddings and retaining those that are sufficiently "close" (under a threshold). Two specific functions, SD (selected dimensions) and MOD-SD (modifier and selected dimensions), are proposed to implement this.

These constraints are integrated into existing compositional methods, particularly focusing on adjective-noun and noun-noun phrases.

Datasets

The authors created a new dataset for their experiments. This dataset consists of bigram noun phrases paired with unigram paraphrase/synonym targets. The creation process involved extracting nouns from WordNet that have bigram phrase synonyms in their synsets. They also expanded upon Turney's (2012) dataset. To ensure the data reflects compositional phrases, they applied filtering heuristics based on WordNet glosses and hypernym relationships. After filtering, 1914 pairs remained. A portion of this dataset was used for tuning the proposed functions, and the rest for evaluation.

Additionally, they evaluated their methods on the noun-noun dataset created by Zanzotto et al. (2010) (referred to as ZZ).

Key Insights and Ideas

  • Semantics over Syntax: The most significant insight is that semantic contribution, rather than syntactic role, is a more crucial factor for accurate phrase composition. This challenges traditional linguistic assumptions and highlights the power of semantic similarity in understanding phrase meaning.
  • Dimensional Relevance: The idea of selecting specific dimensions from modifier embeddings based on their relevance to the head embedding is a novel way to refine composition. It suggests that not all information in an embedding is equally important for a given composition.
  • Simple is Powerful: The paper demonstrates that these linguistically informed constraints can significantly boost the performance of simple compositional methods (like addition and averaging), even outperforming complex neural network architectures. This suggests that focusing on linguistic intuition can be more effective than brute-force deep learning for certain tasks.
  • Generalizability: Preliminary experiments suggest that these constraints can be extended to other phrase types, such as verb phrases, indicating their potential for broader application.

Key Results

The proposed methods, particularly when incorporating the semantic contribution constraint (+Const1), consistently outperformed baselines and other approaches across most evaluation tasks.

  • The sd and mod-sd functions (incorporating both constraints) performed statistically best in 5 out of 6 evaluation tasks, outperforming standard baselines, a neural network, and the Zanzotto et al. approach.
  • The semantic contribution constraint (+Const1) alone significantly boosted the performance of addition, average, and combined addition-multiplication operations, making them statistically the best in 5 of the 6 tasks.
  • For the "Sim" (Plain Similarity) task, "Pr@1", "Pr@2", "Pr@3", "OveNei" (Overlapping Neighbors), and "Clus" (Clustering) tasks, the proposed sd and mod-sd methods, along with baselines enhanced with Const1, were statistically the best.
  • The mod-sd function showed a slight edge in the DistSim task, while sd performed better in the Syn task.

Limitations and Future Work

  • Dataset Limitations: While the authors created a new dataset, they acknowledge that their filtering methods for ensuring compositionality might not eliminate all unsuitable pairs. Future work could involve better filtering techniques.
  • Generalization to Other Phrase Types: The current work focuses on nominal phrases. The authors express interest in testing their constraints on other phrase types, such as verb phrases, to assess their generalizability.
  • Semantic Contribution Classifier: The paper mentions preliminary experiments in training a classifier to determine the semantic contribution constraint. They plan to further develop and make this classifier openly available.
  • Sentence-Level Composition: The ultimate goal is to extend this approach to derive representations for whole sentences by iteratively combining constituent phrases.
  • Contextualized Embeddings: The authors note that using state-of-the-art contextualized embeddings (like BERT) could further enhance their methods, as these embeddings can partially disambiguate polysemous words.

Summary updated

BibTeX
@inproceedings{kalouli2019composing,
  author = {Aikaterini-Lida Kalouli and Valeria de Paiva and Richard Crouch},
  title = {{Composing Noun Phrase Vector Representations}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
}
Figure