Leveraging Three Types of Embeddings from Masked Language Models in Idiom Token Classification

Ryosuke Takahashi, Ryohei Sasano, Koichi TakedaNAACL 2022

TL;DR

This paper improves idiom token classification by leveraging three types of embeddings from BERT: standard contextualized embeddings, uncontextualized token embeddings, and masked token embeddings. By combining these, the authors achieve significant accuracy gains on both English and Japanese datasets, demonstrating that richer representations from LLMs can better capture the nuances of idiomatic language.

What Problem Does This Paper Address?

Understanding language often requires distinguishing between literal and idiomatic meanings of words and phrases. For example, "blew the whistle" can mean literally making a sound or figuratively reporting wrongdoing. This ambiguity, known as idiom token classification, is crucial for accurate text understanding. While recent work has shown that contextualized word embeddings from models like BERT are effective for this task, the authors argue that these models contain even more information that can be exploited. They hypothesize that by combining standard contextualized embeddings with uncontextualized token embeddings (representing the word's inherent meaning) and masked token embeddings (representing the meaning inferred from context), they can achieve a more robust idiom classification.

Methods

The core of the proposed method is an "Embed-Encode-Predict" model, which takes various embeddings as input and predicts whether a phrase is idiomatic or literal. The novelty lies in the types of embeddings used:

  • Standard Contextualized Embeddings: These are the typical BERT embeddings that capture word meaning within a specific sentence context.
  • Uncontextualized Token Embeddings: These are the embeddings that are fed into BERT as input. The authors hypothesize these represent the "standard" meaning of a word, independent of its immediate context.
  • Masked Token Embeddings: These are generated by masking the target word in the input sentence and then obtaining BERT's prediction for that masked token. This is thought to capture the meaning inferred from the surrounding context.

The authors combine these three types of embeddings (for both the verb and noun components of a verb-noun idiom) and feed their concatenation into a standard Embed-Encode-Predict architecture. They also explore combinations of these embeddings to understand the contribution of each.

Datasets

The paper uses two datasets for evaluation:

  • English: The VNC-Tokens dataset (Cook et al., 2008), containing sentences with 53 potentially idiomatic verb-noun pairs. The authors use sentences labeled as "I" (idiomatic) or "L" (literal).
  • Japanese: The OpenMWE Corpus (Hashimoto and Kawahara, 2008), containing 146 potentially idiomatic verb-noun pairs. Similar to the English dataset, they select idioms with sufficient idiomatic and literal examples and use sentences labeled "I" or "L".

The study employs a zero-shot setting, meaning classifiers are trained on phrases that do not contain the specific verb or noun of the target phrase, aiming to generalize to unseen idioms.

Key Insights and Ideas

  • Beyond Contextualization: The paper strongly suggests that LLMs like BERT encode more than just contextual meaning. Uncontextualized and masked embeddings offer complementary information that is valuable for tasks like idiom classification.
  • Uncontextualized Embeddings as a Baseline for Meaning: The hypothesis that uncontextualized embeddings represent a word's inherent meaning is supported by the findings. When a phrase is used literally, its contextualized embedding is closer to its uncontextualized counterpart.
  • Masked Embeddings Capture Contextual Inference: The masked token embeddings seem to capture the meaning inferred from the surrounding words, which is also crucial for disambiguation.
  • Leveraging Existing Models: A significant advantage of the proposed method is that it doesn't require training a new language model. It extracts different types of embeddings from an already pre-trained BERT model, making it computationally efficient.

Key Results

The paper reports significant improvements in macro-averaged accuracy for idiom token classification:

  • The proposed model, which combines all three types of embeddings (BERT[vv; vv_t; vv_m; VN; VN_t; UN_m]), achieved the highest accuracy on both English and Japanese datasets.
  • For English, the accuracy reached 0.865, compared to the baseline BERT[vv; vn] at 0.840.
  • For Japanese, the accuracy reached 0.847, compared to the baseline BERT[vv; vn] at 0.823.
  • Statistical significance tests confirmed that the improvements over the baseline BERT model were significant for both languages.
  • Analysis of cosine similarities between embeddings supported the assumption that literal usage leads to embeddings closer to the uncontextualized versions.

Limitations and Future Work

  • Zero-Shot Setting: While the zero-shot setting is a strength for generalization, it's worth noting that performance might differ in fully supervised settings.
  • Future Work: The authors believe their approach of leveraging multiple embedding types from LLMs can be applied to other natural language processing tasks beyond idiom token classification.

Summary updated

BibTeX
@inproceedings{takahashi2022leveraging,
  author = {Ryosuke Takahashi and Ryohei Sasano and Koichi Takeda},
  title = {{Leveraging Three Types of Embeddings from Masked Language Models in Idiom Token Classification}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
}
Figure