Identifying Temporal Trends Based on Perplexity and Clustering: Are We Looking at Language Change?

Sidsel Boldsen, Manex Agirrezabal, Patrizia PaggioACL 2019

TL;DR

This paper proposes a novel data-driven method to identify temporal trends in historical documents by using perplexity scores from language models as a measure of document similarity. By clustering documents based on these perplexity-based distances, the authors aim to uncover patterns of language change over time, offering a more nuanced approach than traditional discrete time binning.

What Problem Does This Paper Address?

Identifying language change over time in historical document collections is a challenging task. Traditional methods often rely on discrete time intervals, which can be arbitrary and may not accurately capture the gradual nature of linguistic evolution. Furthermore, historical documents, especially medieval charters, often lack explicit dating, making their chronological placement crucial for philological interpretation. This paper tackles the problem of automatically identifying temporal trends and potential language change within such collections, aiming to provide a more data-driven and less arbitrary approach.

Methods

The core of the proposed methodology involves two main components:

  • Perplexity as a Measure of Similarity: The authors use recurrent neural network (RNN) language models (specifically LSTMs) trained on individual documents to calculate perplexity. Perplexity, a measure of how well a language model predicts a given text, is used here as a proxy for the "distance" between documents. The intuition is that documents from similar time periods or linguistic stages will have lower perplexity when compared using a model trained on one of them, indicating greater similarity.
  • K-Means Clustering: Once a matrix of perplexity-based distances between all document pairs is computed, the K-Means clustering algorithm is applied. This groups documents that are similar in terms of their perplexity profiles, with the hypothesis that these clusters will correspond to distinct temporal periods or linguistic variations.

The paper also compares RNN-based perplexity with a simpler baseline method using character trigrams.

Datasets

The study uses a dataset of 291 medieval charters from St. Clara Convent in Denmark. These charters span from 1256 to the Reformation (around the 16th century) and are primarily in Latin and Danish, with some Low German and Swedish texts. The dataset was preprocessed by normalizing document length to mitigate variations due to text length.

Key Insights and Ideas

  • Perplexity as a Temporal Indicator: The study demonstrates that perplexity scores derived from RNN language models correlate significantly with the temporal distance between documents, suggesting that these scores can indeed capture temporal linguistic shifts.
  • RNNs Outperform Trigrams: Recurrent Neural Network language models provide a stronger correlation with temporal distance compared to simpler character trigram models, indicating their greater ability to capture nuanced linguistic evolution.
  • Clustering Reveals Groupings: K-Means clustering, applied to the perplexity-based document representations, successfully groups documents, with some clusters showing a discernible temporal distribution.
  • Language Shift is a Major Factor: The correlation between perplexity and time is stronger when considering the shift from Latin to Danish, highlighting that major language changes significantly influence perplexity measures. However, the method also shows potential for identifying subtler changes within a single language.
  • Beyond Pure Language Change: The clustering results suggest that factors other than just language change, such as document genre (e.g., papal letters) or scribe-specific variations, might also influence the perplexity-based groupings.

Key Results

  • The RNN model achieved a Pearson correlation of 0.65 (p < 0.01) between perplexity and temporal distance, significantly higher than the baseline trigram model's 0.49 (p < 0.01).
  • When partitioning the data by language, the correlation for Latin texts was 0.50 (p < 0.01), while for Danish texts it was a weaker 0.20 (p < 0.01).
  • K-Means clustering with k=7 was found to provide a good fit. Visualizations (t-SNE) showed distinct groupings, with some clusters clearly separating Latin documents from Danish/other languages.

Limitations and Future Work

  • Influence of Non-Temporal Factors: The authors acknowledge that factors beyond language change, such as document genre or scribe-specific styles, can influence perplexity and thus the clustering results. This means that clusters may not solely represent language change.
  • Temporal Outliers: Temporal outliers were observed within clusters, indicating that the clustering is not a perfect temporal segmentation.
  • Future Work: The paper suggests further investigation into how other factors can be incorporated to represent temporal trends. Additionally, exploring how these clusters can be used for extrinsic tasks like temporal document classification is proposed, with the idea that cluster distributions might offer a more nuanced temporal representation than discrete bins. More in-depth analysis, potentially in collaboration with philologists, is needed to fully understand what specific textual characteristics the clusters are capturing.

Summary updated

BibTeX
@inproceedings{boldsen2019identifying,
  author = {Sidsel Boldsen and Manex Agirrezabal and Patrizia Paggio},
  title = {{Identifying Temporal Trends Based on Perplexity and Clustering: Are We Looking at Language Change?}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
}
Figure