Abstract
Lexical Semantic Change (LSC) detection is a critical task in historical linguistics and natural language processing. This paper presents the University of Padova's submission to the DIACR-Ita task at EVALITA 2020. We propose a method that treats words as sets of contextualized vectors generated by multilingual BERT (mBERT). By applying an ensemble of geometric and clustering-based distance metrics to these vector sets across different time periods, we aim to identify words that have undergone significant semantic drift. Our approach achieves a 0.67 accuracy, significantly outperforming standard frequency-based baselines.
Introduction
Language is a living entity; the meanings of words like "mouse" or "stream" have shifted dramatically over the decades. Detecting these shifts automatically is essential for lexicography and for maintaining the accuracy of NLP systems on historical texts. This challenge is particularly pronounced for languages like Italian, which may be considered relatively low-resource in the context of diachronic datasets.
Traditional LSC detection methods often rely on static embeddings (e.g., Word2vec), which collapse all meanings of a word into a single vector. This paper investigates whether the expressive power of contextualized embeddings, specifically from transformers like BERT, can better capture the nuances of words used in varying contexts across time.
Methodology
We formalize the detection of semantic change as a distance-measuring problem between two sets of vectors of varying sizes.
Representation
We utilize multilingual BERT (mBERT) to generate representations. For every occurrence of a target word in a corpus, mBERT produces a unique 768-dimensional vector based on its specific sentence context. Consequently, a word is represented not as a single point, but as a set of vectors $\Phi = \{e_1, e_2, ... e_m\}$.
Distance Metrics
To compare a word's vector set from Time A ($C_1$) to Time B ($C_2$), we evaluated several metrics:
- Average Geometric Distance (AGD): This involves calculating the average Euclidean or Canberra distance between all pairs of vectors across the two sets.
- Hausdorff Distance: A metric measuring the maximum distance from a point in one set to the nearest point in the other, commonly used in spatial analysis.
- Clustering-based Methods: Vectors from both periods are pooled and clustered using K-means or Gaussian Mixture Models. We then use Jensen–Shannon Divergence (JSD) to measure the shift in the distribution of the word across these clusters between the two eras.
Ensemble Approach
Given that semantic change can manifest in various ways (e.g., broadening, narrowing, or sense addition), no single metric is likely to be universal. We employed an ensemble that combines the top-ranked "changed" words identified by each individual metric.
Experimental Setup
The study utilized the DIACR-Ita dataset from EVALITA 2020, consisting of two time-stamped Italian corpora. The task was a "closed" binary classification: models had to predict whether a target word (e.g., palmare, rampante, piovra) had changed meaning using only the provided data.
Results and Discussion
Our proposed ensemble method achieved an accuracy of 0.67, ranking 8th out of 9 teams. While not the top-performing model, it significantly outperformed the frequency baseline (0.50) and the collocation baseline (0.61).
Token vs. Type
A key insight from our work is the "Token vs. Type" struggle. While contextualized embeddings (tokens) are superior for most NLP tasks, they have historically lagged behind static embeddings (types) in semantic change detection. Our results suggest this is not due to a lack of representational power but rather the complexity of comparing high-dimensional "clouds" of vectors.
Sensitivity and Stability
We observed that metrics like Hausdorff distance are highly sensitive to outliers—a single unique context can skew the results. Furthermore, clustering algorithms like K-means exhibited instability based on initialization (seeding) and the choice of $K$, highlighting the need for more robust comparison techniques.
Conclusion
This work demonstrates that contextualized embeddings can effectively detect semantic drift in Italian by treating words as dynamic sets of vectors. Future work will explore supervised multi-lingual frameworks and transfer learning from high-resource languages to further improve detection accuracy in lower-resource settings.
Summary updated