Lexical Semantic Change Annotation with Large Language Models

Thora HagenNAACL 2025

TL;DR

This paper investigates how well state-of-the-art Large Language Models (LLMs) can detect changes in word meaning over time (lexical semantic change) in historical German texts. They found that while some LLMs perform comparably to GPT-4, integrating external historical knowledge through Retrieval-Augmented Generation (RAG) yields mixed results, suggesting LLMs might overemphasize domain differences rather than true semantic shifts.

What Problem Does This Paper Address?

Understanding how word meanings evolve historically is crucial for fields like digital humanities, literary analysis, and historical sentiment analysis. This "lexical semantic change" (LSC) is challenging because historical texts often lack the contemporary data LLMs are typically trained on. While LLMs have shown promise in modern NLP tasks, their ability to interpret nuanced historical language and detect subtle shifts in meaning is still an open question. This paper specifically tackles the task of annotating LSC at the instance level, meaning evaluating how a word's meaning changes within specific contexts in historical documents.

Methods

The researchers evaluated five different LLMs (GPT-40-mini, GPT-40, Llama3.3-70B, Phi-3, and Qwen2.5-72B) on the task of Lexical Semantic Change Annotation (LSCA). They used a zero-shot prompting approach, where the LLM is given a target word and two historical contexts and asked to rate the semantic relatedness of the word in those contexts.

A key aspect of their methodology was exploring the impact of Retrieval-Augmented Generation (RAG). For this, they used a historical German encyclopedia (Meyers' Großes Konversations-Lexikon, 1905) to provide additional historical context to the LLMs. This context was retrieved and integrated into the prompt to see if it improved the models' ability to detect LSC. They also analyzed the results at both a binary level (change vs. no change) and a graded level (four categories of relatedness).

Datasets

The primary dataset used was DURel (Diachronic Usage Relatedness), a manually annotated dataset for German LSC. This dataset contains context pairs for 22 target words, rated by five annotators on a scale of 1 to 4 for relatedness. The contexts were sourced from the DTA corpus (Deutsches Textarchiv), primarily from the 19th century. For the RAG experiments, they utilized Meyers' Großes Konversations-Lexikon, 1905, a historical German encyclopedia.

Key Insights and Ideas

  • LLM Performance Parity: Surprisingly, the Llama3.3 model achieved performance comparable to the much larger GPT-40 model on this task, despite significant parameter differences. This suggests that for historical semantic tasks, model size isn't the only determinant of capability.
  • RAG's Mixed Bag: Integrating historical context via RAG didn't universally improve performance. While it offered slight benefits for smaller models like Phi-3, it actually hindered the performance of larger models like Llama3.3. This implies that larger models might already possess sufficient historical knowledge, and additional context could introduce noise or confusion.
  • Nouns Benefit More from Context: The analysis indicated that RAG tended to benefit nouns more than verbs and adjectives. This could be because encyclopedias often focus on defining concepts (nouns) rather than the nuances of actions (verbs) or descriptions (adjectives).
  • Domain Bias: A significant finding is that LLMs seem to overemphasize domain differences when assessing LSC. For example, a word used in a medical context versus a military context might be flagged as having distantly related meanings, even if the core meaning related to its origin (e.g., "England") remains stable. This bias persisted even with RAG, suggesting LLMs struggle to recognize underlying semantic continuity across different domains.

Key Results

  • Binary LSCA: Figures 1 and 3 show the F1 scores for binary classification. Llama3.3 and GPT-40 performed best, with Llama3.3 showing comparable results to GPT-40. Smaller models like Phi-3 underperformed, even below the majority baseline. RAG showed mixed results, slightly improving Phi-3 but decreasing Llama3.3's performance.
  • Graded LSCA: Figures 2 and 4 illustrate graded LSCA results. Similar trends were observed, with larger models generally outperforming smaller ones. The performance margin between large and small models was more pronounced in the graded task. RAG's impact remained mixed.
  • Error Analysis: The study noted that most errors after RAG were due to higher similarity predictions, particularly affecting words like "feine," "flott," and "packen." Conversely, words like "Kinderstube," "Anstellung," and "Bilanz" were correctly assigned higher similarity scores.

Limitations and Future Work

The authors acknowledge that this is a preliminary study. Key limitations include:

  • Prompt Refinement: The prompts could be further optimized, potentially by incorporating more of the original DURel annotation guidelines.
  • RAG Optimization: The method of retrieving and presenting information for RAG could be improved, as could the preprocessing of encyclopedias and the choice of retrieval strategy.
  • Parameter Tuning: Several parameters (e.g., weighting for embeddings, number of retrieved items) were kept stable for pragmatic reasons but warrant further evaluation.
  • Data Diversity: The observed volatility in F1 scores highlights the need for a larger set of target words and contexts to ensure more reliable model evaluation.

Future work should focus on addressing the LLMs' tendency to overemphasize domain differences and understanding the reasoning behind their predictions. Exploring how different external knowledge sources might influence LSC annotation is also a promising direction.

Summary updated

BibTeX
@inproceedings{hagen2025lexical,
  author = {Thora Hagen},
  title = {{Lexical Semantic Change Annotation with Large Language Models}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
}
Figure