Similarity-Based Cluster Merging for Semantic Change Modeling

Christopher Brückner, Leixin Zhang, Pavel PecinaACL 2024

TL;DR

This paper tackles the challenge of automatically tracking how word meanings evolve over time by proposing a novel cluster merging technique. Their approach significantly improves upon existing methods by intelligently combining clusters of modern word usage, leading to more accurate semantic change detection.

What Problem Does This Paper Address?

Understanding how language changes is a fundamental linguistic and computational problem. Semantic change modeling aims to computationally identify and track shifts in word meanings over historical periods. This involves analyzing large collections of text (corpora) from different time points. A key challenge is that a single word can have multiple meanings (senses), and these senses can drift or new ones can emerge. Existing methods often struggle to accurately group modern word usages with their historical counterparts, especially when a single historical sense might be represented by multiple distinct clusters of modern usage, or when new senses arise. This paper addresses the AXOLOTL-24 Shared Task's Subtask 1, which focuses on unsupervised lexical semantic change modeling, specifically by improving the alignment of modern word usage examples with historical word senses.

Methods

The core of this paper's contribution lies in an improved approach to aligning modern word usage examples with historical word senses. The general pipeline involves:

  1. Embedding: Representing both historical word sense definitions and modern word usage examples as numerical vectors (embeddings) using transformer models.
  2. Clustering: Grouping the modern usage examples into clusters based on their embeddings.
  3. Alignment: Assigning historical word senses to these clusters.

The authors' novel contribution is in refining the alignment step. Instead of a greedy assignment where a historical sense is assigned to the first cluster it matches above a similarity threshold, they introduce Similarity-Based Cluster Merging. This involves:

  • Merging Old Sense Clusters: If multiple clusters of modern usage examples exhibit high similarity to the same historical word sense, these clusters are merged. This helps consolidate related modern usages under a single historical sense.
  • Merging Novel Sense Clusters: After initial assignments, they also propose merging clusters that represent newly induced senses if they are sufficiently similar to each other.
  • Two-Stage Ensembling: They explore ensembling different embedding models and also ensembling multiple clustering runs (using different random seeds) to mitigate the impact of noisy or imbalanced data and to capture different potential clusterings.

Datasets

The research utilizes datasets provided for the AXOLOTL-24 Shared Task, which include:

  • Finnish Corpora: Covering the years 1543-1650 (old period) and 1700-1750 (new period).
  • Russian Corpora: Covering approximately the 19th century (old period) and the years after 1950 (new period).
  • German Test Set: Based on the DWUG dataset, used to evaluate multilingual performance.

Each dataset consists of target words, their historical senses (with definitions and usage examples), and modern usage examples for which sense assignments need to be predicted. The paper notes that the Russian corpus can have noisy or missing old word usage examples.

Key Insights and Ideas

  • Cluster merging is crucial: The most significant insight is that simply assigning clusters to the "best" matching historical sense can be suboptimal. Merging clusters that represent similar modern usages, especially those related to the same historical sense, leads to a more robust alignment.
  • Multilingual embeddings outperform language-specific ones: Surprisingly, general-purpose multilingual sentence embeddings (like LaBSE) often yield better results than language-specific embeddings for this task, suggesting their ability to capture nuanced semantic relationships across languages.
  • Threshold tuning matters: The similarity threshold used for alignment significantly impacts performance, with different optimal thresholds for different languages, highlighting the need for careful parameter selection or cross-validation.
  • Dataset imbalance is a challenge: The quality and balance of word senses and usage examples within datasets can greatly influence results, as seen in the Russian dataset's performance.

Key Results

The paper demonstrates significant improvements over the baseline on the AXOLOTL-24 development sets.

  • Improved ARI and F1: Their cluster merging approach, particularly with LaBSE embeddings, led to substantial gains in both the Adjusted Rand Index (ARI, measuring overall grouping quality) and F1 score (measuring classification of old senses) for Finnish. For Russian, they observed a notable increase in F1 score, though with a slight trade-off in ARI.
  • Ensembling benefits: Ensembling clusterings improved Russian results, while ensembling embedding models showed further gains.
  • Competitive performance: Their final system achieved competitive scores, ranking well on the shared task leaderboards. For instance, on the Finnish test set, their system achieved an ARI of 0.596 and an F1 of 0.756, significantly outperforming the baseline (ARI 0.023, F1 0.230).

Limitations and Future Work

  • Two-stage merging: The authors acknowledge a potential weakness in their two-stage merging process, where old sense disambiguation is prioritized before novel sense induction. They suggest exploring a combined approach or using different similarity thresholds for old and novel sense merging.
  • Dataset comparability: Differences in data quality and balance across languages (e.g., noise in Russian data) can affect comparability.
  • Evaluation scope: The current evaluation doesn't fully account for the absence of recorded word senses in the new period, potentially missing aspects of semantic change.
  • Future directions: They recommend using two different similarity thresholds for old and novel sense cluster merging and suggest the development of better-normalized datasets for improved cross-lingual comparability. They also propose cross-validation as a more robust method for selecting the similarity threshold parameter.

Summary updated

BibTeX
@inproceedings{bruckner2024similarity,
  author = {Christopher Brückner and Leixin Zhang and Pavel Pecina},
  title = {{Similarity-Based Cluster Merging for Semantic Change Modeling}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure