TL;DR
This paper introduces a novel approach to track how word meanings change over time, even in short, noisy datasets like Twitter. By incorporating contextual information about users and their posts, their model significantly improves the accuracy of capturing semantic shifts compared to existing methods, making it a promising tool for understanding language evolution in contemporary digital communication.
What Problem Does This Paper Address?
Understanding how language evolves is crucial for fields like linguistics and digital humanities. Traditionally, this involves analyzing large historical corpora, which are not readily available for recent digital communication platforms like Twitter. Existing methods for tracking semantic change, known as diachronic word embeddings, often struggle with short time spans and noisy data, leading them to primarily capture dominant word senses and miss subtle shifts. This paper tackles the challenge of studying semantic change in a relatively short, five-year Twitter corpus, aiming to develop models that can accurately capture these nuances despite data limitations.
Methods
The authors propose a novel attentional model that builds upon Dynamic Bernoulli Embeddings (DBE). The core innovation lies in conditioning the word embeddings not just on time, but also on a rich set of contextualized features. These features include:
- Spatial: User's geographical region and department.
- Socioeconomic: User's income level derived from French tax data and IRIS datasets.
- Network: Co-mention network derived from user interactions.
- Interest: Topics a user engages with, derived from their posts.
- Knowledge: Features capturing how a user writes.
- Topic: Categories of associated YouTube videos linked in tweets.
They explore three ways to incorporate these features: an unweighted sum, self-attention (allowing the model to weigh features dynamically), and contextual attention (making attention context-dependent). The model is trained using a pseudo log-likelihood objective and optimized with the Adam optimizer.
Datasets
The primary dataset used is a large French Twitter corpus, originally containing 190 million tweets from 2.5 million users between June 2014 and March 2018. For their study, they filtered this to tweets from users located in Metropolitan France, resulting in 18 million tweets from 110,000 users over five years. This dataset was enriched with output from a Stanford parser.
Key Insights and Ideas
- Context is King (Even for Diachronic Embeddings): The paper strongly suggests that incorporating rich contextual information (social, spatial, interest-based) is crucial for accurately capturing semantic shifts, especially in short, noisy datasets. This provides an "inductive bias" that helps models overcome data sparsity.
- Subtle Shifts are Detectable: Unlike previous models that might only capture dominant sense changes, their contextualized approach shows promise in identifying more nuanced semantic evolution.
- Attention Mechanisms Matter: While a simple sum of contextual features can be beneficial, the self-attention and contextual attention mechanisms significantly improve performance, allowing the model to dynamically prioritize relevant contextual signals.
- Twitter as a Valid Corpus: The work demonstrates that even with its inherent noise and short time span, Twitter data can be effectively used to study diachronic semantic change with appropriate modeling.
Key Results
- Improved Fit: The proposed contextual attention model fits the data better than current state-of-the-art dynamic word embedding models.
- Qualitative Success: The model successfully captures subtle semantic shifts, as demonstrated by the example of the word "Bataclan" and the embedding neighborhood of "Emmanuel." It also shows smooth embedding trajectories, comparable to the DBE baseline.
- Quantitative Performance:
- The contextual attention model achieved the best fit in terms of log-likelihood.
- It showed competitive performance in tweet classification tasks (Sentiment Analysis, Topic Categorization) and outperformed baselines in Hashtag Prediction and Conversation Prediction.
- Ablation studies indicated that while the model can work with fewer features, a combination of contextual features is generally beneficial, with "interest" being a particularly important feature.
- Law of Prototypicality: The model's word meaning changes correlated with prototypical exemplars, especially when cluster sizes were small.
Limitations and Future Work
- Static Contextual Features: The authors acknowledge that some contextual features, like social networks, are dynamic and evolving. They suggest that incorporating this temporal aspect could further improve the model's accuracy.
- Feature Importance: While "interest" was highlighted as important, the paper notes that other contextualized features (except interest) were found to be equally important in some ablation studies, suggesting a complex interplay.
- Future Directions: The authors plan to explore the temporal nature of contextual features and investigate more sophisticated ways to model underlying phenomena in dynamic social networks. They also suggest further work on understanding the properties of embeddings that lead to their performance in probing tasks.
Summary updated