A Comparative Study of Embedding Models in Predicting the Compositionality of Multiword Expressions

Navnita Nandakumar, Bahar Salehi, Timothy BaldwinALTA 2018 · OpenReview

Abstract

In this paper, we perform a comparative evaluation of off-the-shelf embedding models over the task of compositionality prediction of multiword expressions ("MWEs"). Our experimental results suggest that character- and document-level models capture knowledge of MWE compositionality and are effective in modelling varying levels of compositionality, with the advantage over word-level models that they do not require token-level identification of MWEs in the training corpus.

Introduction

Multiword expressions (MWEs), such as "couch potato" or "olive oil," present a unique challenge in Natural Language Processing due to their varying degrees of compositionality. While "olive oil" is largely compositional (the meaning is derived from its parts), "couch potato" is idiomatic and opaque. Predicting this degree of compositionality is typically framed as a regression task.

Historically, word-level embeddings have been the standard. However, these models require MWEs to be identified and concatenated (e.g., couch_potato) during training, limiting their flexibility with unseen phrases. This study investigates whether modern character-level and document-level models—which can process arbitrary text on the fly—can compete with traditional word-level approaches.

Methodology

The study evaluates three categories of embedding models:

  1. Word-level: word2vec (skip-gram), where MWEs are pre-tokenized in the training corpus.
  2. Character-level: fastText (using sub-word n-grams) and ELMo (deep contextualized representations, used here without sentence context).
  3. Document-level: doc2vec and InferSent (designed for sentence/paragraph representations).

Performance is assessed using three scoring strategies:

  • Direct Composition: Comparing the MWE vector to the sum of its component vectors via cosine similarity.
  • Paraphrases: Comparing the MWE vector to vectors of its dictionary definitions.
  • Combination: A weighted blend of direct and paraphrase similarities.

Experiments were conducted on two benchmarks: the REDDY dataset and the RAMISCH dataset, both containing 90 English binary noun compounds with human-annotated compositionality scores and paraphrases.

Results and Analysis

The primary metric for evaluation is the Pearson correlation coefficient ($r$) between model predictions and human judgments.

Performance on REDDY Dataset

On the REDDY dataset, word2vec maintained a significant lead, achieving a correlation of $r = 0.710$. Among the alternatives, InferSent emerged as the strongest "off-the-shelf" model ($r = 0.610$), likely due to its training on Natural Language Inference tasks. In contrast, ELMo and doc2vec showed considerably lower performance in the direct composition task.

Pearson correlation results on the REDDY dataset.

Performance on RAMISCH Dataset and the Impact of Paraphrases

The inclusion of paraphrase data (definitions) provided a substantial performance boost across all models. On the RAMISCH dataset, word2vec reached a peak correlation of $r = 0.880$ when combined with paraphrase information. Notably, models that struggled with direct composition, such as ELMo, saw dramatic improvements (jumping from $r = 0.295$ to $r = 0.669$ in the combined setting).

Pearson correlation results on the RAMISCH dataset, illustrating the benefits of paraphrase and combined strategies.

Key Insights

  • The Atomic Advantage: word2vec's superior performance stems from treating MWEs as single atomic tokens, allowing it to capture non-compositional semantics directly.
  • Semantic Grounding: Paraphrases provide critical semantic grounding that is more informative than component words alone.
  • Robustness vs. Accuracy: While character-level models (fastText, ELMo) are currently less accurate, they offer greater robustness as they do not require specific MWE-tokenization during training.

Conclusion and Future Work

While word2vec remains the strongest baseline, document-level models like InferSent show promise as flexible alternatives. Future work should address the "naive" application of contextual models like ELMo by providing proper sentence context and expanding the evaluation to larger, more diverse datasets.

Summary updated

Abstract

In this paper, we perform a comparative evaluation of off-the-shelf embedding models over the task of compositionality prediction of multiword expressions("MWEs"). Our experimental results suggest that character- and document-level models capture knowledge of MWE compositionality and are effective in modelling varying levels of compositionality, with the advantage over word-level models that they do not require token-level identification of MWEs in the training corpus.
BibTeX
@inproceedings{nandakumar2018comparative,
  author = {Navnita Nandakumar and Bahar Salehi and Timothy Baldwin},
  title = {{A Comparative Study of Embedding Models in Predicting the Compositionality of Multiword Expressions}},
  booktitle = {ALTA},
  year = {2018},
  url = {https://openreview.net/forum?id=Xx7KCWy9xp_},
}
Figure