Abstract
Newberry et al. (2017) addressed a fundamental problem in linguistics: testing selective theories of language change against a null model of stochastic drift. By applying the Frequency Increment Test (FIT) from population genetics to diachronic corpora, they suggested that stochasticity plays a larger role in language evolution than previously recognized. This paper replicates their results and explores the sensitivity of the FIT to temporal binning—the process of organizing corpus data into discrete time segments. Through a combination of replication and Wright-Fisher simulations, we demonstrate that while the broad observation of drift holds, results for individual time series are highly sensitive to researcher choices. We conclude that the "researcher degrees of freedom" inherent in applying genetic tests to linguistic data require cautious interpretation and suggest that future work should better integrate the sampling process into evolutionary models.
Introduction
In evolutionary linguistics, a central debate concerns whether language change—such as the regularization of irregular verbs (e.g., spilt to spilled)—is driven by "selection" (a functional or social fitness advantage) or "drift" (random fluctuations in usage). Distinguishing between these two forces is statistically challenging.
A landmark study by Newberry et al. (2017) applied the Frequency Increment Test (FIT), a tool borrowed from population genetics, to 200 years of English verb data. Their findings suggested that most changes are consistent with a null model of drift. However, applying a test designed for genetic alleles to a linguistic corpus involves significant data pre-processing, specifically the grouping of texts into temporal "bins" (e.g., decades or 20-year increments). This study investigates how these arbitrary choices influence the statistical verdict of the FIT.
Methodology
Replication and Sensitivity Analysis
We replicated the original analysis of 36 English verbs undergoing regularization using the Corpus of Historical American English (COHA), which spans 1810–2009 and contains approximately 400 million words. To test the robustness of the FIT, we systematically varied the binning strategies, utilizing both variable-width bins (based on token counts) and fixed-width bins (based on calendar years).
Wright-Fisher Simulations
To establish a "ground truth" where the strength of selection ($s$) is known, we employed Wright-Fisher simulations. We generated thousands of artificial language-change trajectories across varying selection coefficients ($s \in [0, 5]$) and population sizes ($N$). This allowed us to measure how often the FIT correctly identified selection versus drift under different binning conditions and to quantify the rate of false positives.
Key Insights
The Binning Bottleneck
The FIT proved remarkably sensitive to how data is segmented. A specific verb might show a signal of selection when binned by decade but appear as random drift when binned by 20-year increments. This instability suggests that the "researcher degrees of freedom" in data preparation can significantly alter the outcome of the test for individual words.
The S-Curve Conflict
Linguistic change typically follows an S-shaped trajectory (slow initial growth, rapid middle-phase acceleration, and slow final fixation). The FIT, however, assumes a Gaussian approximation of diffusion. Our analysis found that when a change is near the "tails" of the S-curve—where a form is very rare or nearly universal—the FIT often violates its own mathematical assumptions of normality. In our real-world verb analysis, 37% of cases violated these normality assumptions, yet such cases are frequently reported as valid results in the literature.
False Positives and "Dirty" Data
Simulations revealed that binning can artificially smooth out noise, leading the FIT to report selection where none exists. In some scenarios, the false positive rate increased from the expected 5% to over 11%. Furthermore, linguistic data is often "noisy" due to tagging errors. For instance, the verb wed is frequently confused with the abbreviation for Wednesday in older texts, creating phantom frequency shifts that the FIT may misinterpret as evolutionary pressure.
Results and Discussion
Our findings confirm the broad conclusion of Newberry et al.: selection is rarely detected in the regularization of English verbs. Across various binning strategies, only 5 ± 2 verbs out of 36 consistently showed signals of selection. We identified a "Robust Four"—light, smell, sneak, and wake—that maintained a selection signal across almost all tested binning configurations.
However, the FIT appears most reliable only within a specific "sweet spot" of selection strength. If selection is too weak, it is indistinguishable from noise; if it is too strong, the change occurs too rapidly for the test to acquire enough data points for a valid assessment.
Conclusion
While the application of population genetics tools to linguistics is a promising frontier, the Frequency Increment Test should be treated with caution when applied to individual linguistic time series. The sensitivity to temporal binning and the violation of normality assumptions in corpus data suggest that the FIT is better suited for exploratory analysis than as a definitive proof of selection.
Future research should move away from arbitrary pre-processing and toward models that incorporate the corpus sampling process directly into the likelihood function. By accounting for the unique properties of linguistic data—rather than treating it as a direct analog to genetic data—we can develop more robust methods for uncovering the forces that shape language over time.
Summary updated