Diachronic Sense Modeling with Deep Contextualized Word Embeddings: An Ecological View

Renfen Hu, Shen Li, Shichen LiangACL 2019 · DOI

Abstract

Traditional diachronic linguistics often struggles with "meaning conflation deficiency," where a word is represented as a single vector regardless of its multiple meanings. This paper introduces a novel framework that leverages BERT’s contextualized embeddings to track individual word senses over time. By adopting an "ecological" perspective—treating word senses as species within a linguistic ecosystem—the authors provide a fine-grained analysis of how meanings compete, cooperate, and evolve over a 200-year period.

The Challenge of Meaning Conflation

In the study of language evolution, researchers typically use static word embeddings (like Word2Vec) to represent a word's meaning in a specific era. However, these models represent a word as a single point in vector space. If the word gay shifts from meaning "cheerful" to "homosexual," a single-vector model shows the word moving, but it cannot distinguish which specific sense is expanding, which is receding, or how they interact. This paper aims to move from "coarse-grained" word tracking to "fine-grained" sense tracking, revealing the internal dynamics of a word's semantic ecosystem.

Methodology: BERT and the Ecological Lens

The proposed framework consists of two primary stages:

  1. Sense Representation: The authors utilize the Oxford English Dictionary (OED) to establish a ground-truth sense inventory. For each sense, example sentences are processed through a pre-trained BERT model. By averaging the token embeddings from BERT’s final hidden layer, the model creates a distinct "sense embedding" for every definition.
  2. Sense Identification & Tracking: Using the Corpus of Historical American English (COHA), the model generates contextualized embeddings for every instance of a target word across two centuries. It then uses cosine similarity to map each instance to its most likely dictionary sense.

The Ecological View: Senses as Species

To model the resulting data, the authors treat senses like biological species. Using quartic polynomial curve fitting to smooth frequency data, they identify two key behaviors:

  • Sense Competition: When a new or rising sense displaces an existing one.
  • Sense Cooperation: When related senses (e.g., the noun and adjective forms of the same concept) grow in tandem to dominate a word's usage.
Diachronic shift of the word 'gay' showing sense competition and cooperation.

As seen in the case of the word gay, the "light-hearted" sense (Sense 4) dominated the 19th century but began a sharp decline in the mid-20th century as the "homosexual" senses (Senses 2 and 3) rose. This illustrates both competition (Sense 4 vs. Senses 2/3) and cooperation (the simultaneous rise of the adjective and noun forms of the new dominant meaning).

Key Insights and Results

The model was evaluated on both synchronic and diachronic tasks, yielding significant improvements over previous methods:

  • High Precision: The model achieved 93.8% accuracy in matching words in sentences to their correct dictionary definitions.
  • Benchmark Performance: On the Gulordava and Baroni (2011) benchmark for semantic change, the model achieved a Pearson correlation of 0.52, significantly outperforming previous state-of-the-art models which typically reached 0.38.
  • Linguistic Stability: Despite the focus on change, the study found that English is remarkably stable; 70.12% of the 3,220 polysemous words studied maintained a stable dominant sense over the 200-year window.
  • Sense Families: Approximately 31.67% of detected changes involved "sense families"—groups of related meanings (like alien referring to both "foreigners" and "extraterrestrials") that cooperate to take over a word's usage profile.

Conclusion and Future Directions

By moving beyond single-vector representations, this framework allows for a more nuanced understanding of semantic shift. While the method currently depends on existing dictionary inventories, it provides a powerful tool for tracking how subtle nuances evolve. Future work may involve using this ecological framework to predict linguistic trends or assisting lexicographers in identifying when emerging senses have reached a critical mass for inclusion in formal dictionaries.

Summary updated

BibTeX
@inproceedings{hu2019diachronic,
  author = {Renfen Hu and Shen Li and Shichen Liang},
  title = {{Diachronic Sense Modeling with Deep Contextualized Word Embeddings: An Ecological View}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.18653/v1/p19-1379},
}
Figure