Contextualized Embeddings Encode Monolingual and Cross-lingual Knowledge of Idiomaticity

Samin Fakharian, Paul CookACL 2021

TL;DR

This paper introduces a supervised model using contextualized embeddings (like BERT) to distinguish between literal and idiomatic meanings of potentially idiomatic expressions (PIEs). The model demonstrates strong performance not only on known expressions but also generalizes surprisingly well to unseen expressions and even across languages (English and Russian), suggesting these embeddings capture a fundamental understanding of idiomaticity.

What Problem Does This Paper Address?

Many expressions in language have meanings that aren't simply the sum of their parts – they are idiomatic. For example, "hit the road" can mean to physically strike a road, or it can idiomatically mean to "start a journey." This ambiguity, known as idiomaticity, poses a significant challenge for natural language processing (NLP) tasks like machine translation, information retrieval, and sentiment analysis. Accurately identifying whether an expression is used literally or idiomatically is crucial for correct interpretation. While supervised methods have shown promise, they typically require large amounts of annotated data for each specific expression and language. This paper aims to overcome this limitation by leveraging the power of contextualized embeddings to build models that can generalize better, even to expressions and languages they haven't explicitly been trained on.

Methods

The core of this work is a supervised classification model built upon pre-trained contextualized embedding models such as BERT, RoBERTa, mBERT, and RuBERT. For a given sentence containing a potentially idiomatic expression (PIE), the model uses the embedding of the special [CLS] token (or a context window for Russian) as a representation of the sentence's meaning. This representation is then fed into a two-layer neural network to classify the PIE as either idiomatic or literal.

A key aspect of their investigation is evaluating the model's generalization capabilities. They explore:

  • Monolingual experiments: Training and testing on the same language.
  • "Unseen expressions" setup: Training on a subset of PIEs and testing on a held-out set of different PIEs within the same language. This tests if the model learns general patterns of idiomaticity rather than just memorizing specific expressions.
  • Cross-lingual experiments: Training on PIEs in one language (e.g., English) and testing on PIEs in another (e.g., Russian). This is the most ambitious test of generalization, aiming to see if the learned representations can transfer across linguistic boundaries.

They also investigate the impact of incorporating a "canonical form" feature (based on linguistic properties of verb-noun combinations) into the model, a feature that previously showed significant improvements with traditional word embeddings.

Datasets

The paper utilizes existing annotated datasets for PIE idiomaticity:

  • English VNC-Tokens dataset: Contains English verb-noun combinations (VNCs) annotated as literal or idiomatic. This dataset is split into development (EN-DEV) and test (EN-TEST) sets.
  • Russian PIE dataset (Aharodnik et al., 2018): This dataset is broader, including various types of Russian PIEs beyond VNCs, annotated at the token level. For their experiments, they use a subset of this dataset that allows for exact matching between the target expression and the text.

Key Insights and Ideas

  • Contextualized embeddings capture deep idiomatic knowledge: The most significant finding is that models based on contextualized embeddings (BERT, RoBERTa) significantly outperform previous approaches, including those using conventional word embeddings and even linguistically-informed unsupervised methods. This suggests these embeddings encode a richer understanding of idiomaticity.
  • Generalization to unseen expressions is strong: The models show a remarkable ability to correctly classify PIEs that were not part of the training set, indicating they learn generalizable patterns of idiomaticity rather than just memorizing specific phrases.
  • Cross-lingual transfer is surprisingly effective: The models, particularly mBERT, achieve impressive performance when trained on one language (English or Russian) and tested on the other. This performance is comparable to monolingual models tested on unseen expressions, suggesting a shared underlying representation of idiomaticity across languages.
  • "Canonical form" feature offers diminishing returns: Unlike with traditional word embeddings, adding the "canonical form" feature (a linguistic heuristic for VNCs) provides only marginal improvements when used with contextualized embeddings. This implies that the embeddings themselves already capture much of the linguistic information that this feature encodes.
  • Upper layers of BERT encode semantic information: Analysis of different layers of BERT suggests that the upper layers are most effective for this task, aligning with prior research indicating their role in capturing semantic meaning.

Key Results

  • Monolingual (English & Russian):
    • BERT and RoBERTa achieved accuracies in the high 80s to low 90s on the "all expressions" setup for English, outperforming baselines by a significant margin.
  • In the "unseen expressions" setup, BERT and RoBERTa still achieved strong results (e.g., ~79-82% accuracy on EN-TEST), outperforming baselines and showing good generalization.
  • For Russian, RuBERT and mBERT also substantially outperformed the most-frequent class baseline in both "all" and "unseen" expression setups.
  • Cross-lingual:
    • mBERT achieved accuracies around 75.7% (EN-DEV target) and 72.4% (EN-TEST target) for English-to-Russian translation.
  • For Russian-to-English, mBERT achieved 75.2% (EN-DEV target) and 80.1% (EN-TEST target).
  • These cross-lingual results were remarkably close to the monolingual "unseen expressions" performance, highlighting the model's ability to transfer knowledge.

Limitations and Future Work

  • Limitations:
    • The paper acknowledges that the "canonical form" feature, while useful for traditional methods, offers limited added value with contextualized embeddings, suggesting further research might be needed to fully leverage such linguistic knowledge.
  • The cross-lingual experiments were limited to English and Russian, which share a script but have different word orders. The paper notes that performance might differ for languages with significantly different scripts or word orders.
  • For Russian, sentence segmentation was not available, requiring the use of a context window, which might not be optimal.
  • Future Work:
    • The authors plan to expand cross-lingual experiments to include more languages to better understand the impact of augmenting training data with data from multiple source languages.
  • They intend to explore cross-lingual approaches for other MWE prediction tasks, such as predicting noun compound compositionality.
  • Further investigation into optimizing context window sizes for languages lacking sentence segmentation is also suggested.

Summary updated

BibTeX
@inproceedings{fakharian2021contextualized,
  author = {Samin Fakharian and Paul Cook},
  title = {{Contextualized Embeddings Encode Monolingual and Cross-lingual Knowledge of Idiomaticity}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure