Presence or Absence: Are Unknown Word Usages in Dictionaries?

Xianghe Ma, Dominik Schlechtweg, Wei ZhaoACL 2024

TL;DR

This paper tackles the challenge of identifying and defining new word meanings that emerge over time but aren't yet captured in dictionaries. They developed an unsupervised system that maps new word usages to existing dictionary entries and then generates definitions for truly novel meanings, achieving top results in a recent shared task.

What Problem Does This Paper Address?

The evolution of language means new word meanings constantly emerge. While researchers can detect these semantic shifts computationally, a crucial gap exists in knowing whether these new meanings are recognized and documented in dictionaries. Lexicographers traditionally rely on manual, time-consuming processes to update dictionaries. This paper aims to bridge the gap between computational semantic change detection and lexicography by developing a system that can automatically identify word usages with novel meanings and generate dictionary-like definitions for them. This could significantly streamline the process of updating dictionaries.

Methods

The authors propose a two-subtask system for the AXOLOTL-24 shared task.

Subtask 1: Bridging Diachronic Word Usages and Synchronic Dictionaries This subtask focuses on mapping word usages from different time periods to existing dictionary entries. Their approach is unsupervised and involves:

  1. Producing Embeddings: Using m-BERT to encode word usages.
  2. Clustering Embeddings: Employing a novel graph-based clustering method. This method constructs a semantic graph for each target word, with nodes representing average word usage embeddings, cluster centroids, and k-nearest neighbors. This approach is designed to handle low-frequency senses effectively.
  3. Mapping to Dictionary Entries: Aligning dictionary sense entries with the identified clusters. If a cluster's meaning is found in the dictionary, it's mapped. Otherwise, it's flagged as a novel meaning.

Subtask 2: Definition Generation for Novel Word Senses This subtask takes the "unmatched" (novel) word usages identified in Subtask 1 and generates dictionary-like definitions. Their key innovation here is an unsupervised, training-free approach that leverages Large Language Models (LLMs) like GPT-4 and LLaMA-3. Instead of training a model, they prompt these LLMs with specific instructions and examples to generate definitions.

Datasets

The research utilizes datasets from the AXOLOTL-24 shared task for Finnish, Russian, and German languages. These datasets include:

  • Dictionary entries (headwords, definitions/glosses).
  • Word usages from different time periods.
  • Information about the position of headwords within usages.
  • Time period indicators.

The German dataset is noted as being particularly challenging as it's only available in the test phase, meaning no training or development data was provided for this language.

Key Insights and Ideas

  • LLMs as Lexicographers: The paper demonstrates that powerful LLMs, when prompted appropriately, can act as effective "lexicographers" to generate definitions for novel word meanings, outperforming a supervised baseline.
  • Unsupervised Approach is Viable: The success of their unsupervised system, especially in Subtask 2, highlights the potential of LLMs for tasks traditionally requiring extensive labeled data.
  • Graph-Based Clustering for Low-Frequency Senses: Their novel graph-based clustering method shows promise in identifying nuanced semantic shifts, even when word usages are scarce.
  • BERTScore vs. BLEU for Definition Evaluation: The authors observe that while BLEU might favor lexically similar definitions (potentially due to memorization in supervised models), BERTScore better captures semantic similarity, which is crucial for evaluating LLM-generated definitions that might use different phrasing.

Key Results

  • Subtask 1: Their unsupervised graph-based clustering system significantly outperformed the unsupervised baseline system across all languages (Finnish, Russian, and German) in terms of Adjusted Rand Index (ARI) and macro-F1 scores. They achieved first place in Finnish and German, and second in Russian on the Subtask 2 leaderboard.
  • Subtask 2: Their unsupervised LLM-based system for definition generation considerably outperformed the supervised baseline system in all setups, particularly in BERTScore. They ranked first for Finnish and German, and second for Russian in this subtask.

Limitations and Future Work

  • Dataset Size: The datasets are relatively small, with few usages per headword, which is typical for dictionaries but challenging for robust system development. The authors suggest using such datasets primarily for evaluation and call for larger, more comprehensive datasets.
  • Text Encoder: The use of m-BERT, an older model, is acknowledged. Future work could explore more advanced text encoders like XL-LEXEME for better lexical semantic change capture.
  • Data Contamination: A significant concern is the potential for test sets to be included in the LLMs' pre-training data, which could inflate performance. The authors emphasize the need for future work to develop methods for calculating data contamination rates.
  • LLM Performance Nuances: While GPT-4-turbo showed superiority in some NLP tasks, its advantage over GPT-3.5-turbo in Subtask 2 was not substantial, especially for Russian. Further investigation into LLM performance across languages and tasks is warranted.
  • Categorizing Incorrect Definitions: The authors suggest future work could involve a deeper investigation into and categorization of wrongly generated definitions.

Summary updated

BibTeX
@inproceedings{ma2024presence,
  author = {Xianghe Ma and Dominik Schlechtweg and Wei Zhao},
  title = {{Presence or Absence: Are Unknown Word Usages in Dictionaries?}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure