RoDEval: A Robust Word Sense Disambiguation Evaluation Framework for Large Language Models

Luyang Zhang, Shuaimin Li, Yishuo Li, Kunpeng Kang, Kaiyuan Zhang, Cong Wang, Wenpeng LuEMNLP 2025

TL;DR

This paper introduces RoDEval, a novel evaluation framework designed to rigorously assess the Word Sense Disambiguation (WSD) capabilities of Large Language Models (LLMs). RoDEval addresses the limitations of existing WSD evaluation methods by proposing four new metrics that go beyond simple accuracy, revealing significant shortcomings in current LLMs' ability to truly understand and disambiguate word meanings.

What Problem Does This Paper Address?

Word Sense Disambiguation (WSD) is a crucial task in Natural Language Processing (NLP) that aims to identify the correct meaning of a word in a given context. While Large Language Models (LLMs) have shown promise in this area, existing evaluation methods are insufficient. They often rely on single-task evaluations and classification-based metrics that fail to capture the unique generative nature of LLMs. Furthermore, these methods can be misled by LLM biases (like option-order preference) and are susceptible to data contamination from pre-training. This leaves a gap in our understanding of how well LLMs truly perform WSD and what their limitations are.

Methods

The core contribution of this paper is the RoDEval Framework, which introduces four novel metrics to provide a comprehensive evaluation of LLMs' WSD capabilities:

  • Disambiguation Scope (DS): This metric measures the proportion of words a model can successfully disambiguate. It uses a Differently Ordered Multiple-Choice Question (DOCQ) protocol, where the position of the correct answer is varied across trials to mitigate selection bias. A word is only considered successfully disambiguated if the model consistently identifies the correct sense across all positional variations.
  • Disambiguation Robustness (DROS): This metric quantifies how performance varies across different prompt templates and option configurations. It aims to measure the model's stability and resistance to confounding factors.
  • Disambiguation Reliability (DReS): This metric assesses a model's "cognitive awareness" of its own WSD capabilities. It uses a Polysemous Word Cognition Judgment (PWCJ) task, where the model is asked to judge its ability to disambiguate. DReS then measures the alignment between the model's self-assessed confidence and its actual accuracy, identifying instances of overconfidence.
  • Definition Generation Quality Score (DGQS): Recognizing that LLMs are generative, this metric evaluates their ability to produce high-quality sense definitions. It combines semantic similarity with a linguistic richness correction, addressing the limitations of traditional metrics like BLEU and ROUGE, which can unfairly penalize lexical diversity.

Datasets

The paper uses standard English WSD benchmark datasets for evaluation:

  • SemEval2007
  • SemEval2013
  • SemEval2015
  • Senseval2
  • Senseval3

The authors employed a data contamination detection method to ensure the datasets did not pose a risk of contamination for the tested models.

Key Insights and Ideas

  • LLMs struggle with true understanding: Despite impressive performance on some metrics, LLMs often fail to grasp all senses of a polysemous word, leading to incorrect selections not from forgetting, but from incomplete acquisition.
  • Overconfidence is a major issue: LLMs frequently exhibit overconfidence in their WSD predictions, leading to low reliability. This is particularly dangerous as confident incorrect answers can be more misleading than a simple "I don't know."
  • Parameter scaling has diminishing returns for definition generation: While larger models generally perform better, increasing parameters beyond 70 billion shows minimal improvement in the quality of generated definitions.
  • Prompt engineering is critical but can lead to bias: LLMs are highly sensitive to prompt formats, and while careful prompt selection is necessary for evaluation, it can also introduce biases.
  • Fine-tuning for specific tasks can degrade general WSD ability: Models fine-tuned for definition generation tasks show a decline in their core disambiguation capabilities, masking this issue with superficial performance gains.

Key Results

  • The best-performing LLM could only correctly disambiguate 58.2% of polysemous words in test sets.
  • Even the best models had a 25% probability of being confidently incorrect.
  • Models generally exhibit strong robustness (mean DRoS = 0.94 ± 0.05), but still show prompt and selection biases.
  • LLMs with more parameters tend to have weaker biases and stronger robustness.
  • Models with over 70 billion parameters show stable definition generation quality (DGQS > 0.66).

Limitations and Future Work

  • Limited language coverage: The evaluation was conducted on English WSD benchmarks only. The authors note their framework can support custom datasets and diverse languages.
  • Prompt template variations: The analysis of prompt bias used five template variations. Future work could explore more model-specific prompt formulations.
  • Label imbalance in datasets: Existing datasets exhibit label imbalance in polysemous word distributions, which can create fairness issues in MCQA evaluation.
  • Future directions: The authors suggest that making models "self-aware" and improving their ability to handle verbs (which show lower disambiguation accuracy) are important future research directions. They also highlight the need for continued investigation into comprehensively evaluating LLMs' WSD capabilities due to their generative and open-ended nature.

Summary updated

BibTeX
@inproceedings{zhang2025rodeval,
  author = {Luyang Zhang and Shuaimin Li and Yishuo Li and Kunpeng Kang and Kaiyuan Zhang and Cong Wang and Wenpeng Lu},
  title = {{RoDEval: A Robust Word Sense Disambiguation Evaluation Framework for Large Language Models}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure