WordNet under Scrutiny: Dictionary Examples in the Era of Large Language Models

Fatemah Yousef Almeman, Steven Schockaert, Luis Espinosa AnkeCOLING 2024 · OpenReview

TL;DR

This paper scrutinizes WordNet, a foundational lexical resource in NLP, by comparing its dictionary examples to those from other dictionaries and, crucially, to examples generated by ChatGPT. The study reveals that ChatGPT-generated examples significantly outperform WordNet's in terms of informativeness and lead to better word embeddings, suggesting a paradigm shift in how we leverage lexical resources for NLP tasks.

What Problem Does This Paper Address?

Lexical resources like WordNet are crucial for many Natural Language Processing (NLP) tasks, providing definitions and example sentences that help models understand word meanings and usage. However, the quality and effectiveness of these examples, especially in the context of modern NLP and the rise of large language models (LLMs), are not well understood. While guidance exists for traditional uses (like language learning), its impact on NLP systems is unclear. This paper aims to bridge this gap by evaluating the quality of dictionary examples, particularly WordNet's, and comparing them against other resources and LLM-generated content, assessing their utility for downstream NLP applications.

Methods

The researchers employed a multi-faceted approach:

  1. Human Evaluation: Two expert annotators evaluated dictionary examples based on criteria like naturalness, informativeness, and self-containment. They used a questionnaire designed to assess how well examples clarify definitions, especially for difficult terms.
  2. Automatic Metrics: Beyond human judgment, they used automated metrics to assess readability (e.g., Flesch-Kincaid Grade Level) and adherence to "Good Dictionary Examples" (GDEX) criteria, such as sentence fluency and length.
  3. Word Similarity Experiment: To gauge the practical utility of the examples, they generated word embeddings using the MirrorWiC model from dictionary examples. The quality of these embeddings was then evaluated against standard word similarity benchmarks (SimLex-999, SimVerb-3500, SCWS, MEN).

Datasets

The study primarily used examples from:

  • WordNet (WN): The de-facto lexical database for English, widely used in NLP.
  • Oxford Dictionary (CHA): A traditional, well-regarded dictionary.
  • ChatGPT: Examples generated by ChatGPT (gpt-3.5-turbo) using both simple prompts and prompts incorporating GDEX criteria.
  • GPT-3: Examples generated by GPT-3 (text-davinci-003) using similar prompting strategies.

The evaluation also considered other lexical resources like Wikipedia, Wiktionary, Urban Dictionary, CODWOE, and Sci-definition for automatic metric comparisons.

Key Insights and Ideas

  • WordNet's Limitations: WordNet examples, while natural, are often less informative and can be too short or lack fluency, making it harder to learn robust word representations.
  • ChatGPT's Strength: ChatGPT-generated examples are consistently rated as more informative and lead to significantly better word embeddings, suggesting LLMs are becoming a powerful source for high-quality lexical data.
  • GDEX Prompting Nuance: While the GDEX-prompted examples from ChatGPT were longer and more complex, they didn't necessarily outperform simple prompts, indicating ChatGPT might already possess a strong implicit understanding of good example generation.
  • Downstream Performance Matters: The ultimate test is how well examples contribute to NLP tasks. The word similarity experiment strongly supports the superiority of ChatGPT examples for generating useful word embeddings.

Key Results

  • Human Evaluation: ChatGPT examples were judged considerably more informative than those from WordNet and the Oxford Dictionary. WordNet examples were rated as more natural but less informative.
  • Word Embeddings: Word embeddings derived from ChatGPT examples performed considerably better in word similarity benchmarks compared to those from WordNet and the Oxford Dictionary. For instance, Table 5 shows Pearson's Correlation Coefficients for ChatGPT-GDEX reaching up to 0.46 for SimLex and 0.68 for SCWS, significantly higher than WordNet's 0.18 and 0.59 respectively.
  • Automatic Metrics: WordNet examples showed lower sentence fluency and higher penalties for ambiguous words compared to other sources. Wikipedia and Urban Dictionary showed good performance on metrics like sentence length and ambiguity, but higher penalties for non-frequent words.

Limitations and Future Work

The authors acknowledge that the human evaluation involved only two annotators, albeit with expertise. Future work could extend the questionnaire to more resources and LLMs, and leverage the collected scores to train automatic dictionary scoring systems. They also suggest exploring how the intended audience of a dictionary might affect example complexity and utility, and extending the work to other languages.

Summary updated

Abstract

Dictionary definitions play a prominent role in a wide range of NLP tasks, for instance by providing additional context about the meaning of rare and emerging terms. Many dictionaries also provide examples to illustrate the prototypical usage of words, which brings further opportunities for training or enriching NLP models. The intrinsic qualities of dictionaries, and related lexical resources such as glossaries and encyclopedias, are however still not well-understood. While there has been significant work on developing best practices, such guidance has been aimed at traditional usages of dictionaries (e.g. supporting language learners), and it is currently unclear how different quality aspects affect the NLP systems that rely on them. To address this issue, we compare WordNet, the most commonly used lexical resource in NLP, with a variety of dictionaries, as well as with examples that were generated by ChatGPT. Our analysis involves human judgments as well as automatic metrics. We furthermore study the quality of word embeddings derived from dictionary examples, as a proxy for downstream performance. We find that WordNet’s examples lead to lower-quality embeddings than those from the Oxford dictionary. Surprisingly, however, the ChatGPT generated examples were found to be most effective overall.
BibTeX
@inproceedings{almeman2024wordnet,
  author = {Fatemah Yousef Almeman and Steven Schockaert and Luis Espinosa Anke},
  title = {{WordNet under Scrutiny: Dictionary Examples in the Era of Large Language Models}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2024},
  url = {https://openreview.net/forum?id=javeHtfytO},
}
Figure