TL;DR
This paper introduces Word Embedding Networks (WEN), a novel unsupervised method for learning word embeddings that simultaneously align and order them across time slices without prior temporal information. WEN allows for a data-driven investigation into the dynamics of semantic change, revealing that language evolution isn't uniform but rather occurs at varying paces.
What Problem Does This Paper Address?
Understanding how language evolves over time is a fascinating challenge. Traditionally, researchers have looked at word-level changes in language, often assuming a continuous, smooth evolution. However, this assumption might not always hold true, and previous methods for analyzing diachronic (time-based) word embeddings often relied on supervised approaches or assumed a pre-defined temporal order. This paper addresses the need for a more flexible, unsupervised method that can uncover the pace of semantic change by learning the temporal relationships directly from the data, even when the order of time slices isn't explicitly known.
Methods
The core innovation is Word Embedding Networks (WEN). Unlike prior methods, WEN doesn't require temporal information upfront. It learns word embeddings for different time slices and, crucially, simultaneously aligns and orders these embeddings. WEN starts with embeddings that are equidistant and then iteratively adjusts them to reflect semantic relationships.
The method is trained by minimizing a loss function that includes two main terms:
- Reconstruction Term: This term trains the word embeddings to reconstruct a PPMI (Positive Pointwise Mutual Information) matrix, which captures word co-occurrence statistics.
- Sparsity Term: This term encourages sparse vectorial representations for the embeddings.
A key aspect is the dynamic weighting of connections between embeddings from different time slices. This weighting term, $w_{t,t'}$, is updated based on the distance between embeddings, strengthening connections for embeddings that are closer in the vector space. This process allows WEN to infer the temporal ordering and the pace of change. The paper also mentions using Adam optimization with a customized learning rate.
Datasets
The paper evaluates WEN on two distinct datasets:
- New York Times (NYT) 1990-2016: This dataset comprises headlines and lead texts from news articles, totaling approximately 99,872 documents.
- Die Zeit 1947-2017: This dataset includes titles, teaser titles, and teaser texts from a German national weekly newspaper, with a substantial 508,698 news articles.
Both datasets were preprocessed by lemmatization and filtering to include the 20,000 most frequent words that appear in at least three yearly slices.
Key Insights and Ideas
- Time is the Dominant Driver of Semantic Change: The results strongly suggest that temporal proximity is the primary factor influencing changes in word meaning across both datasets.
- Pace of Change is Uneven: Language evolution is not a steady march. WEN reveals distinct periods of faster and slower semantic change, challenging assumptions of uniform evolution.
- Distributional Shifts Matter: Changes in the distribution of topics or sections within a corpus (e.g., more "Opinion" articles in NYT after 2003) can significantly influence the perceived pace of semantic change.
- Unsupervised Temporal Ordering: WEN's ability to learn temporal order without explicit labels is a significant step towards more flexible diachronic language analysis.
- Generalization Across Languages: The model's success on both English (NYT) and German (Die Zeit) datasets demonstrates its potential for cross-lingual applications.
Key Results
- High Neighborhood Accuracy: WEN achieved an 85% "neighborhood accuracy" on the NYT dataset (2003-2016), meaning 85% of the time slices were aligned with their chronological neighbors.
- Solid Accuracy on German Data: For the Die Zeit dataset (1947-2017), WEN achieved 67% accuracy, demonstrating its effectiveness on a non-English corpus not used in parameter tuning.
- Visualization of Change: Laplacian eigenmaps of the learned network weights visually represent the temporal relationships between yearly slices, clearly showing clusters and gaps that correspond to periods of stability and rapid change. For instance, the NYT data shows a circular structure with some years clustered closely (e.g., 2008-2010) and others showing larger gaps (e.g., 2011).
Limitations and Future Work
- Caution with Continuous Change Models: The authors caution against models that strictly assume continuous language change, especially given the observed uneven pace and the influence of distributional shifts.
- Expanding to Unordered Corpora: A promising future direction is to apply WEN to corpora where the temporal order of slices is not known or is inherently unordered (e.g., works grouped by authors). This would allow for discovering proximities between authors based on their language use.
- Investigating Specific Word Changes: While the paper focuses on the overall pace, future work could delve deeper into identifying specific words that undergo significant changes during periods of rapid evolution.
Summary updated