Abstract
Quantifying and predicting morphological productivity is a long-standing challenge in corpus linguistics and psycholinguistics. The same challenge reappears in natural language processing in the context of handling words that were not seen in the training set (out-of-vocabulary, or OOV, words). Prior research showed that a good indicator of the productivity of a morpheme is the number of words involving it that occur exactly once (the hapax legomena). A technical connection was adduced between this result and Good-Turing smoothing, which assigns probability mass to unseen events on the basis of the simplifying assumption that word frequencies are stationary. In a large-scale study of 133 affixes in Wikipedia, we develop evidence that success in fact depends on tapping the frequency range in which the assumptions of Good-Turing are violated.
Introduction
In linguistics, "productivity" describes the capacity of a prefix (e.g., non-) or suffix (e.g., -ness) to generate novel words. Accurately predicting this capacity is essential for Natural Language Processing (NLP) systems tasked with interpreting "Out-of-Vocabulary" (OOV) words. For decades, the count of hapax legomena—words appearing exactly once in a dataset—has served as the gold standard for measuring productivity.
The theoretical underpinning for this measure is often linked to Good-Turing smoothing, which estimates the probability of encountering unseen events based on the frequency of singletons. However, Good-Turing assumes "stationarity"—the premise that word probabilities remain constant over time. This study investigates whether hapax-based measures remain effective in massive, modern datasets like Wikipedia, where topic shifts and specialized jargon frequently violate stationarity.
Methodology and Data
The study analyzed 133 affixes (68 prefixes and 65 suffixes) extracted from a 1.24-billion-word Wikipedia dump (August 2013). To evaluate the robustness of hapax-based metrics, the authors employed several novel techniques:
- The "Far Tail" Ground Truth: Productivity was defined by the type count of words appearing 2 to 11 times in the full corpus.
- Pseudohapaxes: To simulate smaller corpora, "pseudohapax" sets were created—words whose frequency in the full corpus suggests they would have been hapaxes in a smaller sample (e.g., 0.25% to 4% of the total).
- Regression Analysis: Ordinary Least Squares (OLS) regression was used to determine how well the log-count of pseudohapaxes predicted the log-count of words in the "far tail."
The affix list was carefully curated to ensure "decomposability," meaning the removal of an affix must result in a valid, higher-frequency stem.
Key Insights: The Importance of Rare Words
A primary finding is that predicting productivity depends entirely on accessing the frequency range of extremely rare words. Words with frequencies exceeding 0.5 per million (occurring more than 600 times in the corpus) provide negligible information regarding an affix's productivity.
Furthermore, the median frequency of words that successfully predict productivity is remarkably low—often below the threshold of common human knowledge. This suggests that morphological productivity is driven by the underlying "machinery" of language rather than just familiar vocabulary.
Crucially, the hapax measure performs best precisely where the stationarity assumptions of Good-Turing smoothing fail. Productivity is fueled by "bursty," topic-specific words, such as technical terminology or proper names, rather than a constant probability of word creation. Each affix exhibits its own Zipfian rank-frequency distribution; highly productive affixes like non- possess "heavy tails," appearing in a vast array of rare combinations.
Results
The regression analysis demonstrated high predictive power. Pseudohapax sets corresponding to nominal corpus sizes of 2% and 4% (approximately 25M to 50M words) were excellent predictors of the far tail, with $R^2$ values exceeding 0.85.
The study also found a consistent proportionality: the regression slope remained close to 1.0, indicating that the number of types in the far tail is roughly proportional to the number of hapaxes across different affixes. Remarkably, even a small random sample of 3 million words (0.25% of the total) could successfully predict the morphological profile of a corpus 400 times larger, demonstrating significant scale invariance.
Limitations and Future Directions
While the results are robust, certain suffixes like +ville appear highly productive in Wikipedia due to the prevalence of proper names, which may not align with "cognitive" productivity in the same way as -ness. Additionally, the social dynamics of Wikipedia—where different editor groups focus on specific niches like chemistry or sports—likely influence these patterns. Future research could investigate how these sub-communities contribute to lexical growth. Finally, the exclusion of words appearing only once in the full 1.24B corpus (to avoid typos) might omit some genuine, ultra-rare productive forms.
Summary updated