Abstract
Computational models of semantic change have become increasingly sophisticated, yet their alignment with the lived experience of speech communities is rarely tested. This paper, presented at EMNLP 2023, evaluates state-of-the-art NLP models against the perceptions of 15 English speakers in Montreal. Focusing on Quebec English—a dialect shaped by intense contact with French—the study reveals that while models can identify broad shifts, they often miss the social nuances, such as regional pride and perceived "interference," that determine a word's acceptability.
Introduction: The Gap Between Silicon and Society
The field of computational linguistics has developed powerful tools for detecting semantic change—how word meanings evolve over time or vary across regions. Typically, these models are trained on massive, anonymous corpora like Google Books or Twitter. However, a fundamental question remains: do the "changes" these models detect actually resonate with the people using the language?
In Quebec, English exists in a unique sociolinguistic environment, frequently borrowing meanings from French homographs (e.g., using boutique to mean a general "store" or exposition for an "art gallery"). This study investigates whether model-derived scores for these contact-induced shifts align with human acceptability and the social values assigned to them by the local community.
Methodology: Modeling and Human Ground Truth
The researchers compared two primary computational approaches with qualitative human data:
- Type-level Models ($cos_{AVG}$): Using word2vec (skip-gram) trained on regional subcorpora (Montreal vs. Toronto/Vancouver), the authors measured semantic change as the cosine distance between aligned vector spaces.
- Token-level Models ($clust_{CONT}$): Using BERT to generate contextualized embeddings, specific word occurrences were clustered via affinity propagation. This allowed the researchers to calculate the proportion of contact-related senses and regional specificity.
- Human Ground Truth: The "gold standard" was derived from sociolinguistic interviews with 15 Montrealers. Participants rated the acceptability of 40 target words (e.g., terrace, entourage) used in real-world tweets and provided qualitative commentary on their usage.
Key Insights: Frequency and Social Value
A significant finding of the study is the persistent "Frequency Bias" in computational models. As shown in the correlation table below, models often conflate high frequency with semantic stability.
For humans, the relationship is often the opposite: a highly frequent "shifted" word, like terrace (used for patio), is often the most accepted because it has become a standard regionalism. Models, however, tend to assign lower change scores to frequent words ($\rho = -0.764$), potentially overlooking the most successful semantic shifts.
Furthermore, models are blind to the social value of language. Qualitative data showed that speakers distinguish between "localisms" (markers of Montreal identity) and "interferences" (perceived errors due to French influence). Two words might show identical mathematical shifts in a vector space, but one is embraced while the other is rejected.
Results: Aligning Models with Perception
The study found that token-level models generally align better with human perception than type-level models. While type-level models measure the amplitude of a shift, token-level models better capture its diffusion across different contexts.
As illustrated in Figure 1, there is only a weak negative correlation ($\rho = -0.370$) between type-level scores and human acceptability. This suggests that mathematical distance in a vector space is a poor proxy for linguistic naturalness. In contrast, the token-level approach ($clust_{CONT}$) showed a more promising trend, likely because it accounts for the variety of senses a word can inhabit.
Conclusion and Future Directions
This research highlights the necessity of grounding NLP models in sociolinguistic reality. While computational models are excellent at processing vast amounts of data, they currently lack the ability to account for the social dynamics of language contact. Future work may benefit from using multilingual models (like mBERT) to better capture cross-linguistic dynamics and expanding the participant pool to ensure broader statistical robustness. Ultimately, understanding semantic change requires looking beyond the vectors and into the community.
Summary updated