TL;DR
This paper investigates whether Large Language Models (LLMs) truly understand word senses, a crucial aspect of language comprehension. The authors find that while top LLMs like GPT-4o and DeepSeek-V3 perform comparably to specialized systems on traditional Word Sense Disambiguation (WSD) tasks, they excel when allowed to explain word meanings freely, achieving near-human accuracy.
What Problem Does This Paper Address?
Lexical ambiguity, where a single word can have multiple meanings depending on context, is a fundamental challenge in natural language. Effectively resolving this ambiguity is vital for accurate language understanding and generation in NLP applications. While LLMs have shown impressive performance across various tasks, their ability to truly grasp word senses – to disambiguate them correctly – remains underexplored. Existing research has sometimes shown LLMs struggling with infrequent senses or relying on superficial cues. This paper aims to bridge this gap by systematically evaluating LLMs' WSD capabilities and their ability to explain word senses in unconstrained generative settings.
Methods
The authors employ a two-pronged approach:
- Word Sense Disambiguation (WSD) Evaluation: They evaluate a wide range of instruction-tuned LLMs (both open and closed-weight) on four WSD benchmarks. This includes comparing their performance against state-of-the-art supervised WSD systems and human annotators. They also investigate the impact of different prompt templates on LLM performance.
- Generative Evaluation: The top-performing LLMs (GPT-4o and DeepSeek-V3) are evaluated on their ability to understand word senses in three generative tasks:
- Definition Generation: Generating dictionary-style definitions for a word in context.
- Free-form Explanation: Explaining the meaning of a word in context in their own words.
- Example Generation: Creating sentences that use the target word with the same meaning as in a given context. Human annotators are used to evaluate the quality of these generated outputs.
Datasets
The paper utilizes several datasets for evaluation:
- SemEval-2007 dataset: Used for prompt design and initial evaluation.
- Maru et al. 2022 WSD dataset: A manually refined version of popular WSD datasets, serving as the primary test set.
- 42D dataset: A dataset with instances sampled from 42 diverse domains, used to test robustness across domains.
- hardEN dataset: A subset of challenging cases where supervised WSD models failed, used to test robustness on difficult examples.
- WikiPortal WSD dataset: A newly curated dataset of 1,000 manually annotated sentences from news sources, designed to mitigate potential test set contamination.
Key Insights and Ideas
- LLMs are catching up to specialized WSD systems: Top LLMs are now performing on par with, and sometimes surpassing, dedicated WSD models on standard benchmarks.
- LLMs are more robust across domains: Unlike specialized systems that can falter on out-of-domain data, LLMs show better adaptability and performance across diverse datasets.
- Free generation unlocks semantic potential: When LLMs are not constrained by predefined sense inventories and can freely explain word meanings, their accuracy and understanding significantly improve, approaching human levels.
- Prompt sensitivity matters, especially for smaller models: The way a prompt is phrased can significantly impact LLM performance, particularly for smaller models, highlighting the need for careful prompt engineering.
- Positional bias is a potential LLM limitation: Some LLMs show a tendency to favor definitions that appear earlier in a list, suggesting they might rely on superficial heuristics rather than purely semantic appropriateness.
Key Results
- WSD Performance: GPT-4o and DeepSeek-V3 achieve F1 scores comparable to state-of-the-art supervised WSD systems like ConSeC and ESCHER on the Maru et al. 2022 dataset (e.g., GPT-4o scores 82.5%, ConSeC scores 83.0%).
- Challenging Datasets: On the hardEN dataset, where supervised models fail, LLMs like GPT-40 achieve scores around 45.6%, demonstrating their ability to handle difficult cases. On the 42D dataset, GPT-40 achieves an F1 score of 76.8%.
- Generative Tasks: In free-form explanation, LLMs achieve up to 98% accuracy, with GPT-40 outperforming Llama-3.3-70B-Instruct on the 42D dataset (97% vs. 83%).
- Definition Generation: GPT-40 achieves around 90.25% satisfactory outputs on the Maru et al. 2022 dataset, outperforming Llama-3.3-70B-Instruct (85%).
- Example Generation: LLMs successfully generate semantically appropriate sentences 60-70% of the time, though they sometimes struggle with syntactic variety.
Limitations and Future Work
- English-centric: All experiments were conducted solely in English, and evaluating LLMs in other languages is left for future work, noting the scarcity of annotated data and incomplete sense inventories in many languages.
- Limited LLM Coverage: While a wide range of LLMs were tested, more models could be included in future studies.
- Manual Evaluation: The generative component was evaluated manually, which, while avoiding automated metric issues, limits reproducibility. The authors release human annotations to facilitate future automated evaluation approaches.
- Prompt Sensitivity: The study explored various prompt strategies, but further research into adaptive and task-specific prompting techniques is suggested.
- Human vs. LLM Performance: Despite impressive LLM performance, a gap remains between LLMs and human annotators, particularly in disambiguating hard and long-tail senses. Future work could explore how to further close this gap.
Summary updated