The Corpora They Are a-Changing: a Case Study in Italian Newspapers

Pierpaolo Basile, Annalina Caputo, Tommaso Caselli, Pierluigi Cassotti, Rossella VarvaraACL 2021

TL;DR

This paper investigates the reliability of lexical semantic change (LSC) benchmarks and the robustness of LSC detection models when applied to different corpora. The authors find that LSC benchmarks can be biased by the specific corpus used for their creation, and that factors like OCR quality and named entities significantly impact model performance.

What Problem Does This Paper Address?

Understanding how word meanings evolve over time (lexical semantic change, or LSC) is a crucial area in linguistics and natural language processing. Researchers have developed computational models to automatically detect these changes, often relying on large collections of text (corpora) from different historical periods. However, the creation of evaluation datasets (benchmarks) for LSC is a complex process, and their reliability is often tied to the specific corpus used. This raises questions about whether models trained on one corpus can accurately detect LSC in another, and whether the benchmarks themselves are truly representative of language change. This paper addresses this by testing the robustness of LSC detection models and the reliability of existing benchmarks when faced with different source corpora.

Methods

The authors employ a re-implementation of the Word2Vec Skipgram model with Orthogonal Procrustes (OP-SGNS), a state-of-the-art approach for LSC detection. This model is grounded in the Distributional Hypothesis, which posits that words with similar meanings have similar distributions in text. They train and evaluate this model on two distinct Italian newspaper corpora: "L'Unità" and "La Stampa." To assess benchmark reliability, they compare the model's performance on a pre-existing LSC benchmark (DIACR-Ita) when trained on each corpus. They also analyze disagreements between the model's predictions and the benchmark annotations, manually inspecting instances where the model and benchmark diverge. Furthermore, they extend their analysis to the entire common vocabulary of the two corpora to test the model's robustness beyond the benchmark.

Datasets

  • L'Unità Corpus: A diachronic corpus spanning 1945-2014, collected and pre-processed for the DIACR-Ita task.
  • La Stampa Corpus: A diachronic corpus spanning 1945-2005, obtained by crawling a publicly available digital archive.
  • DIACR-Ita Benchmark: A dataset for Italian LSC detection, manually created using the "L'Unità" corpus, containing 18 target lemmas, 6 of which are identified as instances of LSC.

Key Insights and Ideas

  • Corpus Bias in Benchmarks: The study highlights that LSC benchmarks, even when manually created, can be biased by the specific characteristics of the corpus used. For example, the "L'Unità" corpus's lack of advertisements may have obscured the occurrence of a specific sense of the word "ape" (bee/vehicle), leading to its misclassification as an LSC.
  • Sensitivity to Data Quality: OCR errors and the presence of named entities (NEs), especially those that are homonyms with common nouns, are significant sources of noise and can lead to false positives in LSC detection.
  • Model Robustness is Not Guaranteed: While the OP-SGNS model shows some robustness across corpora, its performance can be affected by differences in corpus characteristics and data quality. The accuracy of LSC detection can vary significantly depending on the source material.
  • Social and Cultural Dimensions Matter: The paper emphasizes that social and cultural factors embedded within corpora can influence perceived semantic change, suggesting that these dimensions need careful consideration when creating LSC benchmarks.

Key Results

  • The OP-SGNS model achieved an accuracy of 0.96 ±.02 when tested on the DIACR-Ita benchmark using the "L'Unità" corpus and 0.83 ±.00 when trained on "La Stampa." This difference was partly attributed to the misclassification of words like "ape" and "rampante."
  • A Spearman correlation of 0.67 (p-value < 0.01) was found between the ranked similarities of words across the two corpora, indicating a positive correlation and suggesting that the model's output is generally similar.
  • Manual inspection of 50 sampled lemmas revealed that only 8% (four lemmas) were considered correct examples of LSC. The rest were either false positives due to NEs, OCR errors, or words with a change in usage rather than meaning.

Limitations and Future Work

The authors acknowledge that their preliminary results indicate the need for careful consideration of social and cultural dimensions when creating LSC benchmarks. They also point out that current unsupervised LSC approaches are sensitive to the corpora used and that data quality (OCR, NEs) is a major source of errors. Future work includes extending the analysis to other corpora and languages, as well as exploring different lexical change detection algorithms to confirm their findings. They also suggest the development of more realistic evaluation protocols for unsupervised LSC and research programs focused on acquiring high-quality, distributable diachronic corpora.

Summary updated

BibTeX
@inproceedings{basile2021corpora,
  author = {Pierpaolo Basile and Annalina Caputo and Tommaso Caselli and Pierluigi Cassotti and Rossella Varvara},
  title = {{The Corpora They Are a-Changing: a Case Study in Italian Newspapers}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure