TL;DR
This paper introduces a novel approach to detecting semantic change in words by representing them as Gaussian distributions. By combining learned word embeddings (mean vectors) with frequency-derived variances, the model effectively captures subtle semantic shifts, outperforming methods relying solely on mean vectors, especially for changes in word abstractness.
What Problem Does This Paper Address?
Language is dynamic; words evolve in meaning over time due to cultural and social shifts. Detecting these semantic changes is crucial for understanding language evolution and for applications like information retrieval and historical linguistics. Traditional methods often struggle to capture nuanced shifts, particularly those involving changes in word abstractness (like narrowing or widening of meaning), which don't always manifest as drastic changes in word co-occurrence patterns. This paper tackles the challenge of unsupervised lexical semantic change detection, aiming to identify and quantify these shifts automatically.
Methods
The core of the RIJP team's approach is an enhanced Gaussian embedding model. Instead of learning both the mean vector and covariance matrix for each word (which is computationally expensive and sensitive to initialization, as noted in prior work), they leverage existing techniques. Specifically:
- Mean Vectors: Learned using the standard
word2vecalgorithm. This captures the primary semantic meaning of a word. - Covariance Matrices (Variances): Derived directly from word frequencies within each corpus. This is a key innovation, as it uses readily available frequency information to represent aspects of word meaning that might not be captured by co-occurrence alone, particularly word abstractness.
The semantic shift between two periods (represented by two corpora, $C_1$ and $C_2$) is then measured using the Kullback-Leibler (KL) divergence between the Gaussian distributions of a word in each period. The KL divergence quantifies how much one probability distribution differs from a second, reference probability distribution. A higher KL divergence indicates a greater semantic shift.
Datasets
The paper mentions using corpora $C_1$ and $C_2$ for the SemEval-2020 Task 1. Specific details about these corpora (e.g., their origin, size, or time periods) are not explicitly detailed in the provided text, but they are implied to be suitable for diachronic analysis.
For a preliminary experiment, the BLESS dataset was used to evaluate the model's ability to detect hypernym-hyponym relationships using variances.
Key Insights and Ideas
- Frequency as a Proxy for Abstractness: The paper demonstrates that word frequencies, when used to derive variances in Gaussian embeddings, can effectively capture semantic shifts related to word abstractness (e.g., narrowing or widening of meaning). This is a significant finding, as these types of shifts are often difficult to detect with methods relying solely on co-occurrence patterns.
- Reduced Computational Complexity: By learning only the mean vectors and deriving variances from frequencies, the proposed method significantly reduces computational complexity and sensitivity to initial parameter values compared to traditional Gaussian embedding methods.
- KL Divergence as a Robust Metric: The KL divergence proves to be a suitable metric for measuring semantic shifts between Gaussian distributions, effectively quantifying the difference in word representations across time.
Key Results
The proposed Gaussian-based embedding model achieved competitive results in the SemEval-2020 Task 1:
- Subtask 1 (Classification & Ranking): The model achieved average scores of 0.51 in the evaluation period and 0.70 in the post-evaluation period for Subtask 1.
- Subtask 2 (Ranking): The model achieved scores of 0.14 in the evaluation period and 0.41 in the post-evaluation period for Subtask 2.
- The post-evaluation scores were higher due to appropriate parameter tuning.
- The model demonstrated success in detecting various types of semantic shifts, including narrowing and widening.
- In preliminary experiments, the method using variances derived from word frequencies achieved accuracy comparable to methods using learned variances for hypernym detection.
Limitations and Future Work
- Parameter Tuning: While the proposed method reduces parameter sensitivity compared to traditional Gaussian embeddings, parameter tuning was still necessary to achieve optimal performance, particularly for the evaluation process.
- Focus on Specific Shifts: The paper notes that while the model captures narrowing and widening shifts well, further work is needed to explore other types of semantic shifts more comprehensively.
- Future Work: The authors plan to focus on semantic shift detection across multiple languages and on a more detailed classification of semantic shift types. They also suggest exploring ways to reduce the number of parameters, especially for scenarios where test data is unavailable.
Summary updated