Complexity and Indecision: A Proof-of-Concept Exploration of Lexical Complexity and Lexical Semantic Change

David AlfterACL 2024

TL;DR

This paper explores the connection between how difficult a word is to understand (lexical complexity) and how its meaning changes over time (lexical semantic change). They found that lexical complexity can predict human uncertainty in judging word meaning, suggesting it's a useful feature for detecting semantic shifts.

What Problem Does This Paper Address?

Understanding how language evolves is a key area in NLP. Two related but distinct fields are:

  • Lexical Complexity Prediction (LCP): This aims to determine how difficult a word is to understand in a given context. This is crucial for applications like text simplification, making content more accessible to various audiences.
  • Lexical Semantic Change Detection (LSCD): This tracks how the meaning of words shifts over time. This is important for historical linguistics, understanding cultural evolution, and even for tasks like detecting subtle changes in scientific literature.

While both fields deal with word meaning and are known to be subjective tasks for human annotators, there hasn't been much research exploring a direct link between them. This paper investigates whether the difficulty of a word (lexical complexity) can help us understand or predict changes in its meaning. Specifically, they hypothesize that higher lexical complexity might correlate with greater uncertainty in human judgments of word meaning, and that this complexity can serve as a signal for semantic change.

Methods

The authors propose a proof-of-concept approach to explore the relationship between lexical complexity and semantic change. Their methodology involves:

  1. Lexical Complexity Prediction Model: They fine-tune a single RoBERTa-base model on the SemEval 2021 task data for lexical complexity prediction. Instead of the typical input format that includes genre information, they simplify it to [CLS] word [SEP] sentence to align with the semantic change data. They train this model using the R² objective and early stopping.
  2. Exploring Human Uncertainty: For each target word in the semantic change dataset, they predict its complexity in each of its contexts. They then calculate the average complexity for each time span and the difference in complexity between these spans ($\delta_C$). This $\delta_C$ is then compared to human judgments of semantic closeness, particularly focusing on instances where annotators chose "cannot decide" (label 0).
  3. Predicting Lexical Semantic Change: As a downstream task, they use the fine-tuned lexical complexity model to predict the complexity of words in different contexts across two time spans. They then calculate the correlation between the difference in lexical complexity ($\delta_C$) and the graded lexical semantic change scores obtained from the SemEval 2020 dataset. A baseline RoBERTa-base model (not fine-tuned for complexity) is used for comparison.

Datasets

  • Lexical Complexity Prediction: Data from the SemEval 2021 shared task on lexical complexity prediction. This dataset contains approximately 9000 words from three genres (biblical, parliamentary, medical) manually annotated on a five-point Likert scale for difficulty.
  • Lexical Semantic Change Detection: The English data from the Diachronic Word Usage Graph (DWUG) dataset, used in the SemEval 2020 shared task on unsupervised lexical semantic change detection. This dataset includes 46 target words, each with 200 sentences split across two time spans. Annotators rated the semantic closeness of word pairs on a scale of 1 (unrelated) to 4 (identical), with an option to "cannot decide" (0). Graded lexical semantic change scores (0-1) are derived from these judgments.

Key Insights and Ideas

  • Lexical Complexity as a Proxy for Uncertainty: The study found a strong link between higher lexical complexity differences between sentence pairs and human annotators' tendency to mark "cannot decide." This suggests that when a word's complexity varies significantly across contexts, humans struggle to confidently determine its meaning, leading to indecision.
  • Complexity Difference and Semantic Relatedness: There appears to be an inverse correlation between complexity difference and semantic relatedness. Pairs with larger complexity differences are less likely to be judged as having related senses.
  • Lexical Complexity as a Signal for Semantic Change: While the average complexity within each time span didn't strongly correlate with semantic change, the difference in lexical complexity ($\delta_C$) showed a moderate correlation with graded lexical semantic change. This is a key finding, suggesting that shifts in word difficulty can indeed indicate shifts in meaning.

Key Results

  • Lexical Complexity Prediction Performance: The fine-tuned RoBERTa-base model achieved an MSE of 0.0078 and an R² of 0.524 on the test set, which is considered acceptable for the scope of this study, though slightly lower than the best model from 2021 (MSE 0.0061, R² 0.621).
  • Human Uncertainty: Figure 2 clearly shows that label 0 ("cannot decide") has the highest rank counts when complexity differences are high, while label 4 ("identical") is ranked last, indicating that low complexity differences correspond to same senses.
  • Lexical Semantic Change Detection: The difference in lexical complexity ($\delta_C$) achieved a Spearman's $\rho$ of 0.444 for graded lexical semantic change detection. This result is notable as it outperforms the best baseline from the SemEval 2020 shared task (0.422) and is reasonably close to the state-of-the-art (0.757).

Limitations and Future Work

  • English-Centric Data: The study is limited to English due to data availability. Extending to other languages would be beneficial but requires compiling new lexical complexity resources.
  • Noun-Dominated Datasets: Both datasets primarily consist of nouns. The findings might not generalize directly to other parts of speech.
  • Model Choice: Only a single RoBERTa-base model was fine-tuned as a proof of concept. Future work should explore a wider variety of models, including larger or multiple models, though this can be computationally expensive.
  • Data Set Size: The current work uses a relatively limited dataset, so results should be interpreted with this in mind.
  • Future Directions:
    • Extend the analysis to other languages (German, Swedish, Latin) by compiling graded lexical resources.
  • Investigate hybrid approaches that incorporate lexical complexity prediction as a feature for lexical semantic change detection.

Summary updated

BibTeX
@inproceedings{alfter2024complexity,
  author = {David Alfter},
  title = {{Complexity and Indecision: A Proof-of-Concept Exploration of Lexical Complexity and Lexical Semantic Change}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure