Exploring the Representation of Word Meanings in Context: A Case Study on Homonymy and Synonymy

Marcos GarciaACL 2021

TL;DR

This paper introduces a new multilingual dataset and a systematic evaluation of how modern language models represent word meanings in context, specifically focusing on homonymy (same word, different meanings) and synonymy (different words, same meaning). The study reveals that while Transformer-based models excel at disambiguating homonyms using context, they struggle to differentiate between senses of words when the surrounding sentences are very similar.

What Problem Does This Paper Address?

Understanding word meaning in context is a fundamental challenge in Natural Language Processing (NLP). Traditional methods often represent words with static vectors, meaning a single word form always maps to the same vector, regardless of its usage. While this is simple, it fails to capture the nuances of words with multiple meanings (polysemy) or words that sound/look alike but mean different things (homonymy). Recent advancements, particularly with contextualized word embeddings (CWEs) from models like BERT, have shown promise in capturing context-dependent meanings. However, there's a lack of systematic, controlled evaluations that specifically probe how these models handle lexical ambiguity, especially homonymy and synonymy, across different languages and under varying contextual conditions. Existing datasets often focus on polysemy or use highly constrained phrases, making it difficult to isolate the impact of specific linguistic phenomena.

Methods

The researchers developed a novel multilingual dataset designed to isolate and evaluate the representation of homonymy and synonymy. This dataset contains sentence triples where two sentences use a word with the same meaning (either the same word or a synonym) and a third sentence uses the same word form but with a different meaning. This structure allows for controlled experiments by varying context, word overlap, and part-of-speech tags.

They evaluated two main types of models:

  • Static Embeddings: Specifically, skip-gram fastText models.
  • Contextualized Embeddings (CWEs): This included both multilingual models (mBERT, XLM-RoBERTa, DistilBERT) and monolingual models (BERT for English, Portuguese, and Spanish, and custom-trained BERT for Galician).

The evaluation involved obtaining vector representations for target words in different contexts and then calculating cosine similarities between these vectors. A model was considered successful if it correctly identified the pair of words with the same sense as being more similar than the word with a different sense.

Datasets

The core contribution is a new multilingual dataset for evaluating homonymy and synonymy in context.

  • Languages: Galician, Portuguese, English, and Spanish.
  • Size: Over 3,000 evaluation items.
  • Structure: Sentence triples designed to control for context, word overlap, and POS-tag.
  • Key Feature: Focuses specifically on homonymy (same word form, different meanings) and synonymy (different word forms, same meaning), unlike many existing datasets that lean heavily on polysemy.

Key Insights and Ideas

  • Context is King (Mostly): Transformer-based models (like BERT) are remarkably good at using surrounding text to disambiguate homonyms, correctly identifying when a word like "coach" refers to a bus versus a trainer.
  • Similarity is Tricky: When sentences are very similar, even advanced models can struggle to distinguish between different meanings of the same word, suggesting they might over-rely on superficial sentence structure rather than deep semantic understanding.
  • Homonymy vs. Polysemy: The study highlights that homonyms (unrelated meanings of the same word form) are treated differently by models than polysemous words (related meanings). Models seem to have a stronger consensus on representing homonymous senses.
  • Language-Specific Nuances: While Transformer models perform well across languages, there are slight variations, indicating that language-specific training data and model architectures can play a role.

Key Results

  • Homonymy Detection: The best Transformer models achieved accuracies above 0.98 in some cases (e.g., English Exp1), with an average accuracy of 0.78 across languages for homonymy detection (Exp1). This indicates a strong ability to differentiate unrelated meanings when context is sufficiently varied.
  • Synonymy vs. Homonymy: In experiments comparing synonyms and homonyms, models showed mixed results. While BERT performed better than static models, its accuracy was around 0.67, suggesting that distinguishing between a synonym and a homonym in context remains challenging.
  • Contextualization Effects: Analysis across model layers showed that contextualization benefits generally increase with deeper layers for Transformer models, particularly for tasks heavily reliant on context (Exp1, Exp2). However, in scenarios with very similar contexts (Exp4), static models sometimes outperformed contextualized ones.

Limitations and Future Work

  • Similar Contexts: A key limitation identified is the difficulty models face when distinguishing between different senses of a word when the surrounding sentences are highly similar. This suggests that models might be incorporating "too much" context, leading to confusion.
  • Future Work: The authors plan to extend their dataset to include multiword expressions and less transparent contexts of homonymy. They also aim to systematically explore how multilingual models handle homonymy and synonymy in cross-lingual scenarios.

Summary updated

BibTeX
@inproceedings{garcia2021exploring,
  author = {Marcos Garcia},
  title = {{Exploring the Representation of Word Meanings in Context: A Case Study on Homonymy and Synonymy}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure