TL;DR
This paper introduces "edge induction" as a method to automatically infer missing semantic relatedness scores between word usages in a Word Usage Graph (WUG). By augmenting human-annotated WUGs with these induced edges, the authors demonstrate improved word sense clustering, making it more feasible to create high-quality WUGs with less human annotation effort.
What Problem Does This Paper Address?
Computational linguistics often grapples with understanding how word meanings change over time (Lexical Semantic Change or LSC). A promising approach involves Word Usage Graphs (WUGs), where word usages are nodes and edges represent human-judged semantic relatedness. Clustering these graphs can reveal word senses. However, creating dense, human-annotated WUGs is labor-intensive, leading to sparse graphs that can yield lower-quality sense clusters. This paper addresses the challenge of reducing the annotation burden required for high-quality WUGs by exploring whether automatically inferring missing edge weights can improve the clustering results.
Methods
The core of this work is edge induction, which aims to predict missing edge weights in a WUG based on existing human annotations. The authors explore two main types of features for their induction models:
- Structural Features: These leverage the existing graph structure. A key example is the "log-triangle" feature, which infers the relationship between two usages (u, v) by examining paths of length two through intermediate usages (u, w) and (w, v).
- Contextual Features: These utilize the textual content of the word usages themselves. The paper employs
xl-lexeme-cos, a model based on XLM-R embeddings, which is state-of-the-art for the Word-in-Context (WiC) task.
They experiment with four-way logistic regression models combining these features, trained at different levels of stratification: word-level, language-level, and cross-lingual. They also investigate "iterated inference," where predictions are made in multiple rounds, to see if this improves results. For clustering, they use Correlation Clustering and two variants of the Stochastic Block Model (SBM-binomial and SBM-layers).
Datasets
The experiments utilize existing datasets:
- DWUG_DE dataset: A German dataset for diachronic word usage graphs, specifically the subset annotated with usage-sense information.
- Resampled dataset: A larger dataset of WUGs from German, English, and Swedish, used for evaluating edge induction performance and clustering robustness.
The paper focuses on using existing WUGs and their associated human annotations, rather than introducing new benchmarks.
Key Insights and Ideas
- Edge induction is a viable strategy: The paper strongly suggests that automatically inferring missing edge weights can significantly improve the quality of word sense clusters derived from WUGs.
- Both structural and contextual features matter: Combining graph-based structural information with text-based contextual features yields the best edge induction performance. This is important as it allows for flexibility, especially if researchers want to avoid potential biases from language models.
- Higher-level stratification is beneficial: Training edge induction models at the language or cross-lingual level, rather than just word-level, leads to more efficient inference, especially in low-data scenarios. This leverages shared patterns across words and languages.
- Iterated inference can be helpful (with caveats): While iterated inference can improve performance for language-level and cross-lingual models, word-level models can suffer from error propagation.
- Clustering benefits from enriched graphs: Even with imperfect induced edges, enriching sparse WUGs generally leads to better clustering results compared to using only the original, sparse human annotations.
Key Results
- Improved Correlation: Edge induction models, particularly those combining structural and contextual features (
log-triangle+xl-lexeme-cos), achieved weighted average Spearman correlations with human annotations that were competitive with human-human agreement in some cases, and showed decent agreement even in low-data scenarios. - Clustering Enhancement: Experiment 3 demonstrated that enriching sparsely annotated WUGs with induced edges led to clear improvements in the Adjusted Rand Index (ARI) when comparing clusters to human-annotated senses, across all tested clustering algorithms. For instance, using induced edges generally resulted in higher ARI scores compared to using only ground-truth edges.
- Model Performance: The
log-triangle+xl-lexeme-cosmodel consistently performed best, especially at higher levels of stratification. SBM-binomial generally outperformed correlation clustering when using induced edges.
Limitations and Future Work
- Language Bias: The current investigation is limited to Germanic languages (English, German, Swedish). While the methods are language-agnostic, cross-lingual results should be interpreted with this in mind.
- Sense Inventory Assumption: The evaluation using usage-sense annotation assumes a fixed sense inventory. The authors acknowledge that WUGs are intended to discover unattested senses, and this evaluation method might not fully capture the discovery of entirely new senses.
- Future Work: The authors suggest exploring more advanced approaches for iterated inference, such as Message Passing Neural Networks. They also point to the possibility of joint induction of edges and clusters using more sophisticated Stochastic Block Model variants, though current implementations don't support weighted networks. Investigating strategies that maximize the utility of each human annotation is also highlighted.
Summary updated