Outta Control: Laws of Semantic Change and Inherent Biases in Word Representation Models

Haim Dubossarsky, Daphna Weinshall, Eitan GrossmanEMNLP 2017 · OpenReview

Introduction

In recent years, the field of computational linguistics has proposed several "laws" of semantic change—mathematical rules intended to predict how word meanings evolve over time. These include the Law of Conformity (frequent words change more slowly), the Law of Innovation (polysemous words change faster), and the Law of Prototypicality (prototypical words change more slowly). However, Dubossarsky et al. (2017) challenge these findings, suggesting that these "laws" may be artifacts of the word representation models themselves rather than genuine linguistic phenomena.

The Problem of Model Bias

The primary issue identified is the lack of a control condition in previous studies. When a model like SVD or PPMI reports a change in a word's vector representation between two time periods, it is unclear whether this reflects a true semantic shift or inherent noise in the model's handling of specific word types, particularly those with low frequency.

Methodology: The Null Hypothesis

To isolate model bias, the researchers introduced a rigorous "null hypothesis" test using two types of control corpora:

  1. Chronologically Shuffled Corpus: 5-grams from 1900–1999 were randomly shuffled across decades. In this "fake" history, no real semantic change can occur, as the context of any given word remains statistically identical across all time periods.
  2. Synchronous Subsampling: Multiple random subsamples were taken from a single year (1999) to measure the "change" reported by models between identical time periods.

The study evaluated three common word representations: raw co-occurrence counts, Pointwise Mutual Information (PPMI), and Singular Value Decomposition (SVD). Semantic change was measured using the cosine distance between a word's vector in consecutive decades.

Inherent Model Biases

The results from the shuffled corpus were striking. Despite the absence of any real semantic change, the models consistently reported that frequent words "changed" less than infrequent ones. This demonstrates that the observed correlation between frequency and change is largely a mathematical byproduct of how cosine distance interacts with word frequency.

Correlations between log frequency and log change scores in a shuffled control corpus for Counts (a), PPMI (b), and SVD (c) models.

As shown in Figure 1, the raw counts and SVD-based embeddings exhibit a strong inherent negative correlation between frequency and "change" scores (r = -0.915 and r = -0.793, respectively). Interestingly, while PPMI reduces this bias (r = -0.295), the application of SVD—a standard technique for creating dense embeddings—significantly reintroduces it.

Deconstructing the "Laws" of Language Change

The researchers' analysis suggests that several proposed laws are largely spurious:

  • Law of Conformity: In the genuine corpus, frequency explained 67% of the variance in change. However, in the shuffled control, it still explained 56%. This suggests that the vast majority of the "Law of Conformity" is actually model noise.
  • Law of Innovation: The idea that polysemy correlates with change disappeared when frequency was controlled for. Polysemous words are often simply frequent words; the models were reacting to frequency, not the number of meanings.
  • Law of Prototypicality: This was the only law that maintained a statistically significant effect beyond the control condition, although the effect was considerably weaker than previously claimed.
Distortion correlation with frequency, illustrating how model representations are inherently biased by word frequency.

The authors provided analytical proofs showing that the expected value of cosine distance between two samples of the same word is strictly governed by its frequency-dependent variance. This distortion, visualized in Figure 3, confirms that low-frequency words appear to move more erratically in the vector space due to noise, not semantic evolution.

Conclusion

The findings of Dubossarsky et al. serve as a cautionary tale for computational linguistics. They highlight the danger of interpreting model-specific artifacts as universal linguistic laws. The paper concludes with a call for higher methodological standards: any newly proposed "law" of language must be demonstrated to be significantly stronger in real data than it is in a randomized control condition. Future work must focus on developing truly "bias-free" representations to accurately capture the nuances of semantic change.

Summary updated

BibTeX
@inproceedings{dubossarsky2017outta,
  author = {Haim Dubossarsky and Daphna Weinshall and Eitan Grossman},
  title = {{Outta Control: Laws of Semantic Change and Inherent Biases in Word Representation Models}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2017},
  url = {https://openreview.net/forum?id=Sk-mfGMubr},
}
Figure