TL;DR
This paper introduces "infinite mixture chaining," a novel probabilistic framework for understanding how word meanings evolve and expand over time. By framing this process as a trade-off between representational accuracy and memory complexity, the authors unify existing models and propose a more flexible approach that better captures historical semantic change and human judgments.
What Problem Does This Paper Address?
The lexicon, our vocabulary, is a finite system used to express an infinite number of concepts. This necessitates that words acquire new meanings over time – a phenomenon known as semantic change or word meaning extension. Researchers have observed that this extension isn't random; new meanings often "chain" to existing ones that are semantically similar. However, existing computational models of this "semantic chaining" have largely relied on either prototype theory (representing a concept by its average or typical example) or exemplar theory (representing a concept by all its specific examples). These approaches have limitations: prototype models are too simplistic, while exemplar models are computationally expensive and memory-intensive. This paper aims to bridge this gap by proposing a unified framework that can account for various forms of semantic chaining, offering a more balanced approach to representational accuracy and memory efficiency.
Methods
The core of the paper is the infinite mixture chaining (inf-mix) framework. This probabilistic model views word meaning extension as a temporal prediction problem. Given a new word usage (e.g., "store password"), the model predicts which existing words (e.g., "store") are likely to extend their meaning to encompass this new usage.
The novelty lies in framing this as an efficiency-based trade-off:
- Representational Accuracy: How well the model captures the nuances and dynamic changes of word meaning.
- Memory Complexity: The computational cost of storing and processing these meanings.
The inf-mix model uses a Gaussian Mixture Model approach, where word usages are clustered. The key is that the number of clusters (and thus the memory complexity) is not fixed but can vary, allowing the model to adapt. This is achieved through a Dirichlet Process (DP)-Means clustering algorithm, a non-parametric method that can infer an appropriate number of clusters.
By adjusting a parameter ($\lambda$), the framework can smoothly transition between:
- Exemplar models: When $\lambda \to 0$, each usage becomes its own cluster, maximizing accuracy but also memory.
- Prototype models: When $\lambda \to \infty$, all usages are grouped into a single cluster, minimizing memory but sacrificing accuracy.
- Infinite Mixture models: For intermediate $\lambda$ values, the model finds a balance, creating a flexible number of clusters.
Datasets
The authors evaluate their framework on several real-world datasets:
- English Verb-Noun Compositions: Historical data from the Google Syntactic N-grams corpus, spanning 1850-2000, focusing on verb-object relationships.
- English Adjective-Noun Compositions: Similar to the verb data, but focusing on adjective-noun modifications from the same corpus and time period.
- Chinese Numeral Classifiers: Historical data of Chinese classifiers and the nouns they modify, covering the period 1940-2003.
- Human Judgment of Lexical Semantic Change: The SemEval-2020 Task 1 dataset, which contains human annotations of semantic change for English words across different time periods.
Key Insights and Ideas
- Cognitive Efficiency as a Unifying Principle: The paper elegantly demonstrates that different semantic chaining models (prototype, exemplar, and others) can be seen as different points on a spectrum of cognitive efficiency, balancing accuracy and memory.
- Flexibility of Infinite Mixtures: The inf-mix model's ability to dynamically adjust the number of semantic clusters is crucial. It can represent fine-grained distinctions when needed (like exemplars) but also abstract concepts when memory is a constraint (like prototypes).
- Beyond Dichotomies: The work moves beyond the often-contrasting views of prototype vs. exemplar theories, offering a more nuanced and unified perspective on how word meanings are represented and extended.
- Predicting Historical Change: The framework successfully reconstructs historical semantic shifts in various word classes and languages, suggesting its power in understanding language evolution.
- Human Judgment Alignment: The model's ability to correlate with human judgments of semantic change provides strong evidence for its psychological plausibility.
Key Results
- Balanced Performance: In reconstructing historical semantic change for English verbs, adjectives, and Chinese classifiers, the inf-mix model achieved predictive accuracy comparable to the best-performing exemplar models but with significantly less memory complexity (e.g., requiring ~25x less storage for English verbs).
- Superior to Baselines: Both inf-mix and exemplar models consistently outperformed simpler prototype and 1nn (1-nearest neighbor) models, as well as frequency and random baselines.
- Human Judgment Correlation: The inf-mix model showed a strong correlation (Spearman's $\rho = 0.266$) with human judgments of lexical semantic change, outperforming prototype and exemplar models in this task.
- Optimal Trade-off: The study identified optimal $\lambda$ values for different datasets, demonstrating how the model finds the best balance between accuracy and memory for specific linguistic phenomena. For instance, $\lambda=0.15$ was optimal for human judgment correlation.
Limitations and Future Work
- Simplified Problem Scope: The authors acknowledge that their current model focuses on predicting novel noun-argument pairings. However, word meaning can change in ways not directly tied to new compositions (e.g., metaphorical extensions not based on direct semantic similarity). Future work could integrate mechanisms for analogy and conceptual metaphor.
- Contextual Nuances: The model currently relies on word embeddings and doesn't fully capture highly contextual or idiomatic meaning shifts (e.g., "that person is sick" having different meanings based on context).
- Population-Level vs. Individual-Level: The current framework models word meaning extension at a population level. Extending it to explain individual-level word usage and acquisition, especially in children, is a promising future direction.
- Sociolinguistic Factors: The model doesn't account for broader sociolinguistic factors that drive language change, such as technological innovation or socio-cultural shifts, which could be incorporated in future research.
- Rare/Unconventional Changes: The reliance on historical corpus data might limit the model's ability to explain rare or unconventional word meaning extensions that may not be well-represented in text. Investigating these might require diachronic analysis of linguistic communities.
Summary updated