TL;DR
This paper investigates the reliability of neural word embeddings, a popular tool in Digital Humanities for tracking semantic change. They find that despite good accuracy on standard tasks, the neighborhoods of words in these embedding spaces are often inconsistent, casting doubt on their suitability for drawing firm conclusions about language evolution.
What Problem Does This Paper Address?
In recent years, neural word embeddings (like those from word2vec) have become a powerful tool for computational linguistics and Digital Humanities. These embeddings represent words as vectors in a high-dimensional space, where proximity often signifies semantic similarity. Researchers use these embeddings to track how word meanings change over time (diachronic analysis) by observing how a word's "neighborhood" in the embedding space shifts. However, the reliability of these neighborhood comparisons is crucial for drawing accurate conclusions. This paper addresses the question of whether these neighborhoods are stable and consistent enough to be trusted for such analyses, especially when dealing with historical texts and different languages.
Methods
The authors employ a rigorous experimental setup to assess the reliability and accuracy of neural word embeddings. They train multiple word embedding models (specifically, skip-gram with negative sampling (SGNS) and skip-gram hierarchical softmax (SGHS)) on historical and contemporary English and German text corpora.
- Reliability Measurement: They define reliability as the degree of agreement between the top-N nearest neighbors of a word across multiple independently trained models. A Jaccard coefficient is used to quantify this overlap.
- Accuracy Measurement: They evaluate accuracy using two standard methods:
- Analogy Tasks: Measuring how well the embeddings can solve word analogy problems (e.g., "king" is to "queen" as "man" is to "woman").
- Word Similarity Tasks: Calculating the Spearman rank correlation between embedding-derived similarity scores and human judgments for word pairs.
- Experimental Variations: They systematically vary parameters like the training algorithm (SGNS vs. SGHS), the number of training epochs, and the influence of word frequency and ambiguity.
Datasets
The study uses the Google Books Ngram corpus (GBN) for both English and German. Specifically:
- English Fiction: A subset of the GBN.
- German: Orthographically normalized German texts from the GBN.
Two time spans are used:
- Historical: 1900–1904
- Contemporary: 2005–2009
Key Insights and Ideas
- Neighborhoods are Unreliable: The most striking finding is that even with high accuracy on standard tasks, the word neighborhoods in embedding spaces are often inconsistent across different model runs. This means that what a word is "close to" in the embedding space can vary significantly, making it difficult to draw firm conclusions about semantic evolution.
- SGNS Outperforms SGHS for Reliability: While both methods show similar accuracy, SGNS consistently yields more reliable neighborhoods, especially with more training epochs.
- More Epochs = More Reliability (Up to a Point): Increasing the number of training epochs generally improves reliability, but the gains diminish, and for some cases, accuracy can even slightly decrease after a certain point, suggesting potential overfitting.
- Word Frequency Matters: High-frequency words tend to have more stable neighborhoods, which is expected. However, the paper also notes that medium-frequency words can be particularly problematic for reliability.
- Ambiguity's Counter-Intuitive Effect: For English, higher word ambiguity (more senses) surprisingly correlates with lower reliability, which the authors suggest might be due to infrequent words with fewer contexts. German, however, seems less affected by ambiguity.
- Historical vs. Contemporary: Accuracy is generally higher for the contemporary time span (2005-2009) compared to the historical one (1900-1904), likely due to the test sets being more aligned with modern language.
Key Results
- Top-1 Reliability: While accuracy scores are often in the 0.45-0.60 range, top-1 reliability scores are significantly lower, frequently below 0.40, and sometimes as low as 0.11. This highlights the gap between performance on standard evaluation tasks and the stability of learned representations.
- SGNS vs. SGHS: With 10 training epochs, SGNS achieves higher reliability (e.g., 0.51 for English 1900-1904) than SGHS (e.g., 0.45 for the same scenario).
- German Normalization: Normalizing German orthography slightly decreases reliability but increases accuracy.
- Example of Unreliability: The paper provides an example of the word "romantic," which has high frequency and moderate ambiguity, yet three independently trained models disagreed on its closest neighbor, illustrating the practical implications of unreliable neighborhoods.
Limitations and Future Work
- Test Set Bias: The authors acknowledge that the analogy and similarity test sets are based on modern language and knowledge, which might not perfectly reflect the nuances of historical texts.
- SVDPPMI Potential: They suggest that methods like SVDPPMI, which are not affected by random initialization, might be conceptually superior for diachronic studies, though their own experiments with it were less successful with the chosen hyperparameters.
- Ensemble Methods: Combining multiple identically trained models (an ensemble approach) is proposed as a way to mitigate reliability issues, though this comes at a significant computational cost.
- Hyperparameter Guidelines: Future work could focus on providing better guidelines for selecting hyperparameters for methods like SVDPPMI, especially for complete temporal slices of corpora.
Summary updated