Exploring Word Usage Change with Continuously Evolving Embeddings

Franziska HornACL 2021

TL;DR

This paper introduces a novel method for tracking how word meanings evolve over time by using continuously evolving word embeddings. This approach allows for high-resolution analysis of semantic shifts, enabling researchers to observe changes on a weekly or monthly basis, unlike traditional methods that often require yearly or longer intervals.

What Problem Does This Paper Address?

Languages are dynamic, with words acquiring new meanings or shifting their usage over time. Understanding these semantic shifts is crucial for fields like computational linguistics and digital humanities, especially in fast-paced environments like social media. Traditional methods for analyzing these changes, known as diachronic embeddings, often rely on static word embedding models (which struggle with smaller datasets) or complex dynamic language models. Furthermore, many approaches require splitting corpora into discrete time slices, which can introduce biases by assuming meaning changes only occur at these predefined intervals. This paper aims to provide a simpler, more intuitive, and higher-resolution method for tracking these shifts.

Methods

The core contribution is the development of continuously evolving embeddings. This method leverages pre-trained transformer models (like BERT) to generate contextualized embeddings for each word occurrence. Instead of averaging embeddings over predefined time slices, the paper proposes computing a weighted running average of these contextualized embeddings. This is achieved by updating a global embedding for each word as new sentences are processed chronologically. A weighting parameter, $\alpha$, is introduced to give more importance to recent occurrences, effectively implementing an exponential forgetting of past contexts.

The authors also developed a web application (EvolvEmb App) that allows users to easily explore these continuously evolving embeddings. Users can compute embedding snapshots at user-defined time points and then visualize the evolution of nearest neighbors for specific words, providing an interactive way to explore semantic shifts.

Datasets

The primary dataset used for demonstration and evaluation is a corpus of 95,203 New York Times article snippets published between April 1st, 2019, and December 31st, 2020. This dataset was chosen for its chronological order and its representation of a fast-paced environment.

The paper also mentions the use of a synthetic dataset to demonstrate the method's ability to capture artificially introduced semantic shifts, inspired by previous work.

Key Insights and Ideas

  • High-Resolution Semantic Shift Detection: The ability to take "snapshots" at arbitrary time points allows for semantic shift detection at a resolution of weeks or months, a significant improvement over traditional yearly or multi-year intervals.
  • Simplicity and Intuition: The weighted running average approach is presented as a conceptually simple and intuitive method, making it more accessible than complex dynamic language models.
  • Applicability to Smaller Datasets: By leveraging pre-trained transformer models, the method can be applied to smaller corpora, overcoming a limitation of older static embedding models.
  • No Need for Predefined Time Slices: The chronological processing and running average eliminate the bias introduced by arbitrary time slice definitions.
  • Interactive Exploration: The EvolvEmb App provides a user-friendly interface for visualizing these evolving embeddings, making the exploration of word usage change more accessible.

Key Results

  • The method successfully identifies semantic shifts, as demonstrated by the example of "Dorian" becoming synonymous with a devastating hurricane.
  • When applied to shuffled article snippets (to simulate a non-chronological corpus), the resulting semantic shift scores were significantly lower, highlighting the importance of chronological order.
  • Fine-tuning the transformer model (BERT) on the target corpus generally improved the ability to identify semantic shifts, especially for new word usages not present in the original training data.
  • A comparison with traditional SGNS word2vec models trained on discrete time slices showed that while faint patterns were visible, the plots were generally noisier than those produced by the continuously evolving embeddings.

Limitations and Future Work

  • Fine-tuning Recommendation: While pre-trained transformer models are effective, fine-tuning is recommended for clearer identification of semantic shifts, especially for novel word usages.
  • Hyperparameter $\alpha$: The weighting parameter $\alpha$ in the running average influences how frequently snapshots need to be taken. Its optimal setting can vary and might require experimentation.
  • Subword Embeddings: In the synthetic experiment with "presidentcoronavirus," the use of subword embeddings by the transformer introduced a preconception about the word's meaning, which fine-tuning helped to mitigate.
  • Future Work: The authors hope their provided code will enable others to explore word usage change in their own corpora. The method is particularly well-suited for studying fast-paced environments like social media.

Summary updated

BibTeX
@inproceedings{horn2021exploring,
  author = {Franziska Horn},
  title = {{Exploring Word Usage Change with Continuously Evolving Embeddings}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure