Detecting Contact-Induced Semantic Shifts: What Can Embedding-Based Methods Do in Practice?

Filip Miletic, Anne Przewozny-Desriaux, Ludovic TanguyEMNLP 2021 · DOI

Abstract

While word embeddings have shown promise in detecting semantic change, their practical utility in identifying rare, contact-induced shifts remains under-explored. This study evaluates Word2Vec and BERT-based methods on Quebec English, where proximity to French induces specific meaning shifts. We find that while models perform well on curated benchmarks (up to 80% accuracy), they struggle significantly in "discovery" mode due to a low signal-to-noise ratio in social media data. We conclude that these methods are currently best utilized as tools for human-in-the-loop exploration rather than fully automated discovery engines.

Introduction

Linguists have long been interested in using word embeddings to track how word meanings vary across regions (synchronic) or time (diachronic). Most research has focused on "proof-of-concept" studies using well-known examples. This paper tackles a more difficult, real-world descriptive problem: contact-induced semantic shifts. Specifically, it examines Quebec English, where proximity to French often causes English words to take on French-influenced meanings (e.g., using deception to mean "disappointment," mirroring the French déception). The challenge is twofold: these shifts are rare (a "needle in a haystack" problem), and social media data is notoriously noisy.

Methodology

The researchers implemented a two-pronged approach to evaluate embedding performance:

  1. Type-level Embeddings (The Global View): Word2Vec (SGNS) was used to create static embeddings for words in Montreal, Toronto, and Vancouver. To compare them, the authors used Orthogonal Procrustes (OP) and Spatial Referencing (SR) alignment techniques. A "variation score" based on cosine distance was then calculated to identify words that behave differently across regions.
  2. Token-level Embeddings (The Local View): BERT was used to generate contextualized embeddings for individual word occurrences. Affinity Propagation clustering was then applied to group similar usages, allowing researchers to see if specific senses (like the French-influenced one) were unique to a region.

Data and Benchmarking

The study utilized a large-scale Twitter corpus from 2019, containing approximately 35 million tweets and 629 million tokens across Montreal, Toronto, and Vancouver. To provide a rigorous evaluation, the authors introduced a new 80-item test set consisting of 40 known semantic shifts in Quebec English and 40 stable words matched for frequency and part of speech.

Results and Discussion

Performance on Curated Benchmarks

On the balanced 80-item test set, the type-level models demonstrated strong performance. The best configuration—using Orthogonal Procrustes with 100-dimensional vectors and a window size of 5—achieved a peak accuracy of 0.80.

Model accuracy on the 80-item test set across different hyperparameters, showing peak performance of 0.80.

The Discovery Challenge and the "Evaluation Gap"

The most striking insight is the disconnect between test-set performance and real-world utility. When the model was asked to find shifts from the entire vocabulary (discovery mode), the precision for the top 50 candidates was a dismal 0.02. Only one word (exposition) was a true contact-induced shift.

As shown in the variation score distribution, the rare signal of semantic shifts is often drowned out by noise. Most high-ranking candidates were false positives driven by proper names, local topical variation (e.g., real estate trends), or misspellings.

Variation scores across the vocabulary, illustrating the 'evaluation gap' where rare semantic shifts (red x) are often indistinguishable from noise (grey dots) at the top of the rank.

Conclusion

While embedding-based methods are highly accurate on curated benchmarks, fully automated discovery of rare linguistic phenomena remains unfeasible due to the high volume of social media noise. However, these tools are invaluable for accelerating manual data exploration. Token-level clustering, for instance, can turn a month-long manual task into a few hours of work by grouping similar usages for expert review. Future work should focus on developing filters for topical noise to improve discovery precision.

Summary updated

BibTeX
@inproceedings{miletic2021detecting,
  author = {Filip Miletic and Anne Przewozny-Desriaux and Ludovic Tanguy},
  title = {{Detecting Contact-Induced Semantic Shifts: What Can Embedding-Based Methods Do in Practice?}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  doi = {10.18653/v1/2021.emnlp-main.847},
}
Figure