Conceptual Change and Distributional Semantic Models: an Exploratory Study on Pitfalls and Possibilities

Pia Sommerauer, Antske FokkensACL 2019

TL;DR

This paper explores the challenges and potential pitfalls of using distributional semantic models (like word embeddings) to study conceptual change in the digital humanities. The authors demonstrate that seemingly robust findings can be artifacts of the models or data, proposing a set of guidelines to improve the reliability of such studies.

What Problem Does This Paper Address?

The digital humanities increasingly use distributional semantic models (DSMs) to track how word meanings, and by extension concepts, change over time. While these models have shown promise in identifying clear semantic shifts (e.g., "gay"), applying them to complex concepts like "Racism" in a methodologically sound way is challenging. The core problem is that DSMs reflect word usage, not direct conceptual perception, and are susceptible to various biases and random factors. This can lead researchers to mistakenly identify model artifacts as genuine conceptual changes, undermining the validity of their findings. The paper aims to highlight these pitfalls and offer practical advice for more reliable research.

Methods

The authors employ a case study of the concept "Racism" to investigate the reliability of DSMs for diachronic analysis. They:

  • Define Hypotheses: Formulate specific hypotheses about how the semantic relationships between words related to "Racism" (core concepts, subconcepts, instances, related concepts) should have changed over the 20th century, distinguishing between "old" and "new" racism.
  • Use Multiple Models: Compare results from different types of DSMs, including count-based models (PPMI, SVD) and prediction-based models (Skip-gram with Negative Sampling - SGNS).
  • Vary Initializations: For SGNS models, they train multiple models on the same data but with different random initializations to assess the impact of this factor on results.
  • Employ Control Words: Use "control words" that are semantically related but are not expected to exhibit the same changes as the target words, to help distinguish genuine shifts from general data biases.
  • Analyze Nearest Neighbors: Examine the "nearest neighbors" of target words in the semantic space to understand how their contextual associations evolve.

Datasets

The study utilizes two primary corpora:

  • COHA (Corpus of Historical American English): A well-balanced corpus but relatively smaller in size (average 24.5 million words per decade).
  • Google Ngram Corpus: A much larger corpus, but unbalanced and with limitations (e.g., full text access is not always available).

The paper does not introduce new benchmark datasets but rather applies existing methods to these established corpora for the specific use case of "Racism."

Key Insights and Ideas

  • "Naive" approaches can be misleading: Applying standard DSM methods without critical checks can lead to conclusions that are artifacts of the model or data, rather than true conceptual change.
  • Model architecture matters: Different types of DSMs (count-based vs. prediction-based) can yield different results, highlighting the need to understand their inherent biases.
  • Randomness is a significant factor: Prediction-based models are sensitive to random factors like initialization and data order, which can drastically alter results, especially with smaller corpora.
  • Control words are crucial: Using control words is essential for verifying whether observed changes are specific to the concept under study or are broader linguistic phenomena.
  • Conceptual systems are complex: Simply looking at a single word's semantic shift is insufficient; understanding the relationships between multiple words representing a concept is key.

Key Results

  • The "naive" approach, using state-of-the-art methods, confirmed some hypotheses about the shift in "Racism," showing statistically significant shifts in the expected direction for several word pairs.
  • However, when subjected to more rigorous testing (multiple models, different initializations, control words), many of these initial findings proved unstable.
  • For instance, out of 25 nearest neighbors for a word, only 2-5 were shared across all tested models, indicating significant variation.
  • The study found that as much as 14 out of 25 nearest neighbors for "racial" could vary depending on the model's initialization, even when trained on the same data.
  • Only a handful of the initial hypotheses were consistently confirmed across all methods and settings, underscoring the fragility of many findings.

Limitations and Future Work

  • Word selection: The authors acknowledge that their selection of words and relations might not be perfectly representative of the entire conceptual system of "Racism." An interdisciplinary team with deeper expertise might propose more robust hypotheses.
  • Data properties: While control words help, they cannot capture all potential data biases (e.g., balance and size).
  • Unexplored factors: The impact of the order in which word-context pairs are processed in prediction-based models was not fully explored in this paper.
  • Future Work: The authors propose combining their guidelines with experiments that explore shuffled and synchronic corpora (as done by Dubossarsky et al., 2017) to further investigate the distinction between actual changes and artifacts. They emphasize the need for more profound knowledge about how semantic shifts are interpreted to draw careful conclusions about conceptual change.

Summary updated

BibTeX
@inproceedings{sommerauer2019conceptual,
  author = {Pia Sommerauer and Antske Fokkens},
  title = {{Conceptual Change and Distributional Semantic Models: an Exploratory Study on Pitfalls and Possibilities}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
}
Figure