RAW-C: Relatedness of Ambiguous Words in Context (A New Lexical Resource for English)

Sean Trott, Benjamin BergenACL 2021

TL;DR

This paper introduces RAW-C, a novel dataset of human-annotated relatedness judgments for ambiguous words in context. They demonstrate that while current contextualized word embeddings (like BERT and ELMo) can distinguish between same and different senses of a word, they struggle to capture the nuanced, graded relatedness humans perceive, especially between homonyms and polysemes.

What Problem Does This Paper Address?

Understanding word meaning is fundamental to natural language processing (NLP). Words are often ambiguous, meaning they have multiple distinct senses (e.g., "bat" as an animal vs. a piece of sports equipment). Even unambiguous words can evoke subtly different interpretations depending on context. This dynamic and context-dependent nature of meaning poses a significant challenge for NLP models. While recent advancements in contextualized word embeddings (like BERT and ELMo) have improved performance on tasks like Word Sense Disambiguation (WSD), there's a lack of resources that specifically evaluate how well these models capture the graded nature of human semantic understanding – how similar or related different uses of a word feel to humans, even within the same or different senses. This paper aims to bridge this gap by creating a dataset and evaluating current models against human intuitions.

Methods

The authors created the RAW-C dataset by adapting stimuli from existing psycholinguistic studies. They selected 115 ambiguous English words and, for each, generated four sentences: two for each of two distinct meanings. These sentences were carefully constructed to be minimally different, often by changing only a single word, to isolate the effect of the target word's meaning. Lexicographical resources (Merriam-Webster and OED) were used to label the ambiguity type as either homonymy (unrelated meanings) or polysemy (related meanings). Human participants then rated the relatedness of the target word's usage across pairs of these sentences on a Likert scale from "totally unrelated" to "same meaning." They also collected human estimates of sense dominance. The authors then evaluated contextualized embeddings from BERT and ELMo by computing the cosine distance between word representations in different contexts and correlating these distances with human relatedness judgments.

Datasets

  • RAW-C (Relatedness of Ambiguous Words in Context): This is the novel dataset introduced in the paper. It consists of 112 ambiguous words (after excluding some items), with 672 sentence pairs in total. For each word, there are graded human relatedness judgments for sentence pairs representing same-sense and different-sense uses, categorized by ambiguity type (homonymy vs. polysemy).
  • Existing Datasets (for context): The paper reviews several existing datasets for word similarity and WSD, including SimLex-999, SimVerb-3500, WordSim-353, MTurk-771, MEN, Stanford Contextual Word Similarity (SCWS), Word in Context (WiC), and CoSimLex. These are discussed in relation to how they address either disambiguation or contextual gradation, but not necessarily both with the same level of control as RAW-C.

Key Insights and Ideas

  • Human relatedness is graded, not just categorical: Humans perceive a spectrum of relatedness between word uses, which goes beyond simple "same sense" vs. "different sense" distinctions.
  • Contextual embeddings capture basic sense distinctions, but miss nuance: BERT and ELMo can differentiate between same and different senses of a word, but they struggle to capture the fine-grained, graded relatedness that humans perceive, especially for homonyms.
  • Homonymy vs. Polysemy matters to humans, but less so to current models: While humans show distinct patterns of relatedness for homonyms versus polysemes, current models' ability to distinguish between these is marginal.
  • Controlled contexts reveal human sensitivity: Even minimal changes in sentence context can lead to significant differences in human judgments of word relatedness, suggesting humans are highly sensitive to subtle contextual cues.
  • Cosine distance underestimates same-sense similarity and overestimates different-sense homonym similarity: This suggests models might be too "spread out" in their representations for same senses and too "close" for different-sense homonyms, compared to human perception.

Key Results

  • Inter-annotator agreement: The average Spearman's rank correlation between participants and the consensus rating was 0.79, indicating high reliability in human judgments.
  • Model performance vs. human judgments: Cosine distances from BERT and ELMo explained about 37% of the variance in human relatedness judgments.
  • Categorical structure's importance: A model incorporating only "Same Sense" and "Ambiguity Type" explained significantly more variance (R² = 0.66) than the model using only BERT and ELMo's cosine distances.
  • BERT vs. ELMo: BERT performed slightly better than ELMo in correlating cosine distances with human relatedness (Spearman's ρ = -0.58 for BERT, -0.53 for ELMo). However, both models performed considerably worse than human inter-annotator agreement.
  • Systematic model biases: Cosine distance tended to underestimate how related humans find same-sense uses and overestimate how related they find different-sense homonyms.

Limitations and Future Work

  • Limitations of RAW-C:
    • The broad category of "polysemy" could be further subdivided (e.g., metaphor, metonymy) for more granular analysis.
  • The dataset uses experimentally controlled minimal pairs, rather than naturally occurring sentences, which might not fully reflect real-world language use.
  • Future Work:
    • Grounding language models: Incorporating sensorimotor or social-interactional associations into language models could improve their ability to distinguish meanings and modulate them contextually.
  • Lexicon-informed models: Exploring models that explicitly incorporate knowledge about human mental lexicons and sense representations (e.g., SenseBERT) could yield improvements.
  • Finer-grained polysemy annotation: Annotating polysemous word pairs for specific semantic relations (metaphor, metonymy) could help assess which relations correlate with distributional profiles.
  • Augmenting neural models: Investigating ways to augment models like BERT with semantic knowledge or representational schemes to improve performance on tasks like RAW-C.

Summary updated

BibTeX
@inproceedings{trott2021raw,
  author = {Sean Trott and Benjamin Bergen},
  title = {{RAW-C: Relatedness of Ambiguous Words in Context (A New Lexical Resource for English)}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure