Grammar and Meaning: Analysing the Topology of Diachronic Word Embeddings

Yuri Bizzoni, Stefania Degaetano-Ortlieb, Katrin Menzel, Pauline Krielke, Elke TeichACL 2019

TL;DR

This paper uses diachronic word embeddings to analyze how grammatical words, not just content words, change over time in scientific writing. By examining the "topology" of word representations, they reveal that grammatical words evolve differently from lexical words, offering insights into the interplay between grammar and meaning in language change.

What Problem Does This Paper Address?

Understanding how language changes over time is a fundamental challenge in linguistics. While much research has focused on how the meaning of individual words shifts (lexical semantic change), less attention has been paid to how grammatical structures and word usage evolve, particularly in specialized domains like scientific writing. This paper aims to bridge this gap by investigating changes in grammatical word usage within the context of scientific discourse, using computational methods to uncover subtle shifts in style and register.

Methods

The core of this research lies in the application of diachronic word embeddings. These are computational models that represent words as vectors in a multi-dimensional space, where words with similar meanings and usage contexts are located closer together. By training these models on corpora from different time periods, researchers can track how these word representations change over time, revealing shifts in meaning and usage.

Specifically, the authors employ a structured skip-gram model that accounts for word order, which is crucial for capturing grammatical patterns. They then analyze the resulting embedding spaces using:

  • Dimensionality Reduction (t-SNE): To visualize the high-dimensional embedding spaces in a way that highlights relationships between words.
  • Clustering Algorithms (Affinity Propagation, DBSCAN, MiniBatch K-Means): To group words with similar distributional properties, identifying potential semantic or grammatical categories.
  • Relative Entropy (Kullback-Leibler Divergence): To measure how distinctive specific linguistic features (like part-of-speech ngrams) are for certain time periods or word groups.
  • Inner and Outer Distance Metrics: To quantify how words within a group relate to each other (inner distance) and how the group as a whole relates to the rest of the vocabulary (outer distance). This helps to understand whether words are becoming more specialized or more isolated.

Datasets

The primary dataset used is the Royal Society Corpus (RSC), specifically version 4.0. This corpus contains publications from the Philosophical Transactions and Proceedings of the Royal Society of London, spanning from 1665 to 1869. It comprises approximately 32 million tokens and 10,000 documents, with rich metadata including author, publication date, and linguistic annotations. This corpus is well-suited for diachronic analysis due to its historical depth and focus on scientific writing.

Key Insights and Ideas

  • Grammatical Words Behave Differently: The study reveals that grammatical words (like determiners and conjunctions) show distinct patterns of change compared to lexical words. While lexical words tend to become more specialized and spread out in the embedding space, grammatical words exhibit a more "reclusive" tendency, with stable inner distances but increasing outer distances.
  • The "Ing-form" as a Window into Lexis-Grammar Interplay: The paper focuses on "ing-forms" of verbs (e.g., "running," "examining") as they are poly-functional, acting as both gerunds and present participles. This duality makes them ideal for studying the tension between lexical meaning and grammatical function.
  • Semantic Diversification in Scientific Language: The overall trend observed in the embedding space is an expansion, suggesting a process of specialization at the word level. This is particularly evident in scientific writing, where vocabulary becomes more refined and domain-specific over time.
  • Emergence of Academic Verb Clusters: The analysis of "ing-forms" identifies distinct semantic groups, including "academic verbs" (e.g., "ascertaining," "determining"), "change-of-state verbs" (e.g., "saturating," "diluting"), and "motion verbs" (e.g., "passing," "falling"). These clusters become more pronounced over time.
  • Grammatical Roles of "Ing-forms" Shift: The research shows that academic and change-of-state "ing-verbs" are predominantly used as gerunds, while motion "ing-verbs" are more often used as participles. This distinction in grammatical function is captured by the embeddings.

Key Results

  • The study demonstrates a clear increase in both inner and outer distances for lexical words over time, indicating semantic specialization.
  • Function words show a stable inner distance but an increasing outer distance, suggesting they become more distinct from the rest of the lexicon without changing their internal relationships.
  • "Ing-forms" of verbs cluster into three main semantic groups: academic, motion, and change-of-state.
  • Analysis of "ing-forms" reveals that academic and change-of-state verbs are primarily used as gerunds, while motion verbs are used as participles, with this distinction becoming clearer over time.
  • The paper provides visualizations (e.g., Figure 1, 2, 3, 4, 5, 6) illustrating these trends, showing the expansion of the embedding space and the distinct trajectories of different word categories.

Limitations and Future Work

The authors acknowledge that their analysis of "ed-verbs" is based on the "somewhat simplistic assumption" that they represent the majority of past tenses. They also note that multiword expressions and phrases were excluded from their models to maintain agnosticism about corpus content.

For future work, the authors plan to:

  • Investigate other poly-functional words, such as "wh-words," which are also likely involved in the development of scientific style.
  • Analyze the embedding space in terms of domain-specific vocabulary to further explore the trend of specialization in scientific language.
  • Track diversification related to register formation as a high priority for future research.

Summary updated

BibTeX
@inproceedings{bizzoni2019grammar,
  author = {Yuri Bizzoni and Stefania Degaetano-Ortlieb and Katrin Menzel and Pauline Krielke and Elke Teich},
  title = {{Grammar and Meaning: Analysing the Topology of Diachronic Word Embeddings}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
}
Figure