QMUL-SDS @ DIACR-Ita: Evaluating Unsupervised Diachronic Lexical Semantics Classification in Italian

Rabab Alkhalifa, Adam Tsakalidis, Arkaitz Zubiaga, Maria LiakataarXiv 2020 · arXiv

TL;DR

Researchers from Queen Mary University of London and the Alan Turing Institute evaluated several unsupervised methods for detecting how Italian word meanings change over time. Their top-performing system, which utilized Temporal Word Embeddings with a Compass (TWEC) and a Continuous Bag-of-Words (CBOW) architecture, achieved 83.3% accuracy, ranking 3rd in the DIACR-Ita 2020 shared task.

What Problem Does This Paper Address?

Language is not static; the meanings of words shift over decades and centuries—a phenomenon known as diachronic lexical semantics. For example, the word "cell" meant a small room in a monastery long before it meant a biological unit or a mobile phone.

Detecting these shifts automatically is a major challenge in Natural Language Processing (NLP), especially because most research has historically focused on English. The DIACR-Ita 2020 task challenged researchers to build unsupervised systems that could identify which Italian words changed meaning between two distinct time periods, using only raw text corpora and no labeled training data.

Methods

The authors explored a variety of strategies to map word meanings across time. Their approach generally followed a two-step pipeline:

  1. Generating Word Vectors: They used standard architectures like Skip-gram (SG) and Continuous Bag-of-Words (CBOW) to turn words into mathematical vectors. They compared two generation methods:
  2. Independent (IND): Training separate models for each time period.
  3. Compass (TWEC): Training a "base" model on the combined corpora, then fine-tuning it on each period while freezing certain weights to ensure the vector spaces remain aligned.
  4. Alignment and Mapping: For the independent models, the vector spaces are not naturally comparable. The authors tested several alignment techniques:
  5. Orthogonal Procrustes (OP): A classic geometric rotation to align two matrices.
  6. Machine Learning Mappings: Using Linear Regression (LR) and Feed-Forward Neural Networks (FFNN) to learn how to "translate" vectors from one time period to another.
  7. Measuring Change: They used cosine similarity to measure the distance between a word's vector in Period A and its (aligned) vector in Period B. Words with low similarity were flagged as having shifted in meaning.

Datasets

The study utilized two diachronic Italian corpora provided by the DIACR-Ita 2020 organizers, representing two different time periods.

  • Training/Evaluation Sets: The authors used Italian stop-words (common words like "the" or "and" that rarely change meaning) as a ground-truth "stable" set to evaluate how well their alignment methods worked.
  • Test Set: A small, gold-standard set of 18 words (12 stable, 6 shifted) was used for the final ranking.

Key Insights and Ideas

  • Stop-Words as Anchors: One of the most effective strategies was using stop-words to learn the alignment mapping. Because these words are semantically stable, they provide a "cleaner" signal for the alignment process than using all common words, which might include noisy, shifting terms.
  • The Power of the Compass: The TWEC (Compass) approach outperformed independent training followed by alignment. By forcing the models to share a common "compass" (base weights), the resulting vectors were more naturally comparable across time.
  • Distributional Skew as a Quality Signal: The authors found that successful models showed a highly skewed distribution of cosine similarities for stop-words—meaning most stop-words were assigned very high similarity scores.
Figure 1: Frequency of stop words by their cosine similarity scores, where each subfigure pertains to a different model.
  • CBOW > Skip-gram: In almost all experimental configurations, the CBOW architecture yielded better results than Skip-gram for this specific task.

Key Results

The quantitative results, summarized in the table below, highlight the superiority of the TWEC-CBOW model.

Performance metrics across different models and evaluation methods.
  • Top Accuracy: The TWEC_CBOW model achieved the best accuracy of 83.3%.
  • Ranking: This performance earned the team 3rd place in the DIACR-Ita 2020 competition.
  • Outlier Detection: The authors experimented with a variance-based threshold ($\mu - 2\sigma$) for classification. While it helped some weaker models (like Linear Regression) improve, it didn't surpass the mean-based threshold for the top-performing TWEC model.

Limitations and Future Work

The authors noted that the test set provided by the competition was quite small (only 18 words), which can make evaluation sensitive to small changes.

Looking ahead, the team aims to move beyond static embeddings and explore contextualized language models like BERT and ELMo, which can capture more nuanced, sentence-level semantic shifts. They also expressed interest in dynamic graph models to better capture the evolving relationships between words over time.

Summary updated

Abstract

In this paper, we present the results and main findings of our system for the DIACR-ITA 2020 Task. Our system focuses on using variations of training sets and different semantic detection methods. The task involves training, aligning and predicting a word's vector change from two diachronic Italian corpora. We demonstrate that using Temporal Word Embeddings with a Compass C-BOW model is more effective compared to different approaches including Logistic Regression and a Feed Forward Neural Network using accuracy. Our model ranked 3rd with an accuracy of 83.3%.
BibTeX
@misc{alkhalifa2020qmul,
  author = {Rabab Alkhalifa and Adam Tsakalidis and Arkaitz Zubiaga and Maria Liakata},
  title = {{QMUL-SDS @ DIACR-Ita: Evaluating Unsupervised Diachronic Lexical Semantics Classification in Italian}},
  year = {2020},
  eprint = {2011.02935},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2011.02935},
}
Figure