TL;DR
This paper tackles the challenge of Word Sense Disambiguation (WSD) by proposing a novel framework that leverages Large Language Models (LLMs) to create high-quality, large-scale WSD training data. They demonstrate that LLM-generated or annotated data can train compact WSD models that rival or even surpass their LLM teachers, offering a more efficient and generalizable solution than traditional methods.
What Problem Does This Paper Address?
Word Sense Disambiguation (WSD) is a fundamental task in Natural Language Processing (NLP) that aims to identify the correct meaning of a word in a given context. Existing WSD methods heavily rely on manually annotated datasets, which are expensive to create, limited in scale, and often struggle to generalize to new domains or out-of-distribution scenarios. This scarcity of diverse, high-quality training data hinders the development of robust and broadly applicable WSD systems. The paper addresses this by proposing a method to automatically generate or annotate WSD data using LLMs, aiming to overcome the limitations of manual annotation and improve the generalization capabilities of WSD models.
Methods
The core of this work is a knowledge distillation framework that uses LLMs to construct "silver-standard" WSD datasets. They explore two main distillation approaches:
- Generation-based Distillation: LLMs are used to generate diverse example sentences for specific dictionary senses. This involves:
- Decoding-Based Diversity: Varying LLM decoding parameters (temperature, top-k, top-p) to introduce lexical variation.
- Prompt-Based Diversity: Using specialized prompts to guide LLMs to generate examples with specific domain (DomGen) or form/content (DivGen) variations.
- Annotation-based Distillation: LLMs act as automatic annotators, assigning sense labels to polysemous words within real-world corpus sentences. This approach leverages large, open-ended corpora and an iterative annotation process to build a comprehensive dataset.
The generated or annotated datasets are then used to fine-tune compact WSD models, such as ESCHER, which is a BART-based model known for its efficiency and accuracy.
Datasets
- Training Data: The paper introduces two types of LLM-generated/annotated datasets:
- Synthetic Dataset (Dsyn): Generated by LLMs based on lemma-definition pairs from WordNet.
- Annotated Dataset (Danno): Created by LLMs annotating sentences from the British National Corpus (BNC).
- Evaluation Datasets:
- Domain-Specific Test Sets: ALL, ALL_NEW, S10_NEW, SoftEN. These are designed to reflect distributions similar to manually annotated data.
- General-Domain Test Sets: 42D and HardEN. These are intended to challenge models in diverse, unfamiliar, or out-of-distribution contexts. HardEN is highlighted as particularly challenging.
- Existing Datasets: WordNet (used as the sense inventory), SemCor (a commonly used gold-standard WSD training set), and the Brown corpus (from which SemCor is sampled).
Key Insights and Ideas
- LLMs as Data Creators: The paper convincingly demonstrates that LLMs can be powerful tools not just for inference but also for constructing high-quality training data for downstream NLP tasks like WSD.
- Annotation Outperforms Generation (for WSD): While generation-based methods offer diversity, annotation-based distillation generally yields better performance for WSD. This is attributed to the use of natural, real-world contexts and a more direct task for the LLM (identifying meaning vs. generating novel text).
- Sense Coverage is Crucial: Broadening the sense inventory in the training data (achieved through LLM annotation) significantly improves generalization performance, especially on challenging general-domain test sets.
- Compact Models Can Match LLMs: Distilled compact models can achieve performance comparable to, or even exceeding, their much larger LLM teachers, offering a practical solution for deployment.
- Balancing Data is Key: For annotation-based methods, balancing the distribution of senses (e.g., by capping instances per sense) is important for improving general-domain performance, even if it slightly reduces performance on domain-specific sets.
Key Results
- The proposed LLM-distilled models significantly outperform models trained on gold-standard corpora, especially on general-domain benchmarks.
- The annotation-based distilled model achieved a 50% F1 gain on the most challenging test set (HardEN) compared to gold-data-trained models.
- The best distilled models could match or even exceed the performance of their LLM teachers, despite having over 1000 times fewer parameters.
- For example, ESCHER distilled from DeepSeek-v3 annotations achieved an F1 score of 50.00 on HardEN after sense balancing, a substantial improvement.
- Sense coverage from LLM-annotated data was found to be 2.6 times greater than that of SemCor, directly correlating with better generalization.
- The cost of generating synthetic data was less than $100**, while annotation costs ranged from **$50 to $400 for different LLMs, demonstrating cost-effectiveness.
Limitations and Future Work
- Generation Limitations: The diversity strategies for generation, while effective, may still fall short of capturing the full fine-grained variety of real-world contexts. More advanced and controllable generation methods are needed.
- Silver Data Reliability: LLM-generated or annotated data is not always perfectly reliable and can be prone to hallucination or semantic drift, especially with strictly constrained prompts. Post-processing techniques are suggested for improvement.
- Test Set Size: The general-domain test sets used in the study are relatively small, which might limit the comprehensiveness of the evaluation. The authors suggest the need for larger, more representative benchmarks.
- Future Work: The paper suggests exploring more advanced generation methods, developing better techniques for filtering and correcting silver data, and creating larger, more representative benchmarks for general-domain WSD.
Summary updated