When Time Makes Sense: A Historically-Aware Approach to Targeted Sense Disambiguation

Kaspar Beelen, Federico Nanni, Mariona Coll Ardanuy, Kasra Hosseini, Giorgia Tolfo, Barbara McGillivrayACL 2021

TL;DR

This paper introduces "Targeted Sense Disambiguation" (TSD), a novel task for identifying specific senses of words in historical texts. By fine-tuning BERT models on historical corpora and incorporating temporal information, they demonstrate significant improvements in disambiguating word senses over time, crucial for digital humanities research.

What Problem Does This Paper Address?

Natural Language Processing (NLP) models often struggle with the dynamic nature of language. Words and their meanings evolve over time, a phenomenon particularly evident in historical texts. Standard NLP models, trained on contemporary language, fail to capture these historical shifts, leading to inaccuracies when analyzing older documents. This gap is critical for fields like digital history and cultural analytics, where understanding the precise meaning of words in their historical context is paramount. The paper addresses this by developing methods to make NLP models "historically-aware," specifically for the task of determining if a word in a time-stamped text corresponds to a particular, pre-defined sense.

Methods

The core of this work lies in adapting and evaluating language models for historical text. The authors propose two main research questions:

  • RQ1: Historical Language Models: They investigate whether fine-tuning BERT (a powerful transformer-based language model) on historical text corpora improves performance on historical sense disambiguation. They create two historical BERT models: BERT_1850 (trained on books predating 1850) and BERT_1900 (trained on books up to 1900). These are compared against a standard BERT_base model.
  • RQ2: Time-Sensitive Disambiguation: They explore methods that explicitly incorporate temporal information into the disambiguation process. This includes:
  • Sense Embeddings from Transformers: Extracting BERT embeddings for keywords and then using either a binary centroid (representing positive vs. negative classes) or a sense centroid (representing specific senses) for classification.
  • Diachronic Sense Embeddings: Enhancing the sense centroid approach by either filtering training data to include only temporally close examples or weighting examples based on their temporal proximity to the target text.
  • Binary Perceptron: A simpler classifier that directly uses BERT keyword vectors.

They also compare these methods against established baselines like random guessing, Lesk (a dictionary-based method), and a Support Vector Machine (SVM) classifier.

Datasets

The primary dataset used is the Oxford English Dictionary (OED). The OED is invaluable because it provides:

  • A vast collection of senses for each word.
  • Time-stamped quotations illustrating each sense.
  • Definitions and historical usage data.

To augment the OED, they also leverage the Historical Thesaurus of English (HTE) to retrieve synonyms and related senses. For training the historical BERT models, they use a collection of historical books in English digitized by the British Library (referred to as MBL).

Key Insights and Ideas

  • Time is Crucial for Language Understanding: The paper strongly reinforces the idea that language is not static. Ignoring the temporal context leads to significant performance degradation, especially for historical texts.
  • Fine-tuning BERT on Historical Data Works: The experiments show a clear benefit from fine-tuning BERT on historical corpora. This suggests that transformer models can effectively learn and incorporate historical linguistic patterns.
  • Sense Centroid is a Powerful Approach: Among the evaluated methods, the "sense centroid" approach, particularly when combined with historical BERT models, consistently yields the best results. This indicates that representing senses as aggregated embeddings is a robust strategy.
  • Time-Sensitive Methods Help, Especially for Older Texts: While historical models are generally beneficial, explicitly incorporating temporal information (like weighting by time) further boosts performance, particularly for older documents where semantic shifts might be more pronounced or less captured by general historical models.
  • TSD is Pragmatic for Digital Humanities: The authors argue that Targeted Sense Disambiguation is a practical and efficient task for researchers in digital history and cultural analytics, enabling them to track the evolution of specific word senses.

Key Results

  • BERT Outperforms Baselines: Across various time periods and methods, BERT-based approaches consistently outperformed traditional baselines like Lesk and SVM.
  • Historical Models Improve Performance: BERT_1900 generally achieved the highest scores, indicating that fine-tuning on a broad historical corpus is effective. BERT_1850 performed best for earlier periods but its performance dropped faster for more recent data.
  • Sense Centroid Dominates: The "sense centroid" method, especially with historical BERT models, achieved the highest F1 scores, often exceeding 0.60 for the positive class across different time periods. For example, BERT_1850 sense centroid achieved an F1 score of 0.602 for the 1760-1850 period.
  • Time-Sensitive Weighting Offers Gains: In Experiment 2, time-sensitive methods (filtering and weighting) generally improved upon non-time-sensitive approaches, particularly for abstract concepts and when dealing with texts further removed from the language model's training data.

Limitations and Future Work

  • Data Sparsity: The OED, while rich, presents a data sparsity challenge, with a limited number of historical examples for each sense. The paper's data expansion step (using synonyms and related senses) helps mitigate this.
  • Experimental Complexity: The task is inherently difficult due to the complexity of target concepts and the limited historical examples.
  • Future Work: The authors suggest extending this work to track semantic change across multiple historical textual collections. They also highlight the broader applicability of developing NLP methods that can handle diachronic data for fields beyond historical research, such as information retrieval and digital libraries. They plan to further investigate curated case studies to simulate real-world historical research scenarios.

Summary updated

BibTeX
@inproceedings{beelen2021when,
  author = {Kaspar Beelen and Federico Nanni and Mariona Coll Ardanuy and Kasra Hosseini and Giorgia Tolfo and Barbara McGillivray},
  title = {{When Time Makes Sense: A Historically-Aware Approach to Targeted Sense Disambiguation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure