An experimentalist’s perspective on replicability in typology: commentary on “Replication and methodological robustness in quantitative typology” by Becker and Guzmán Naranjo

Kenny SmithLinguistic 2025 · DOI

Abstract: Article An experimentalist’s perspective on replicability in typology: commentary on “Replication and methodological robustness in quantitative typology” by Becker and Guzmán Naranjo was published on July 30, 2025 in the journal Linguistic Typology (volume 0, issue 0).

Introduction

Quantitative typology—the statistical study of patterns across the world’s languages—is currently navigating its own version of the "replication crisis." As researchers increasingly apply sophisticated statistical tools to large-scale cross-linguistic datasets, a troubling trend has emerged: different teams often reach wildly different conclusions using the same data. In a recent study, Becker and Guzmán Naranjo (B&GN) demonstrated that many classic findings in the field are difficult to replicate, raising questions about the robustness of typological science.

In this commentary, Kenny Smith argues that while these findings are sobering, they should not lead to a crisis of faith in the discipline. Instead, he suggests that the inherent limitations of observational typology can be mitigated by adopting an experimentalist’s perspective. By treating typological patterns as "hypothesis generators" for controlled experiments, the field can move beyond identifying correlations toward proving the causal mechanisms that shape human language.

The Problem: The Garden of Forking Paths

The primary obstacle to replicability in typology is what Smith identifies as the "garden of forking paths." When handling complex, messy linguistic data, researchers must make a series of micro-decisions: how to clean the data, which variables to include, how to handle phylogenetic relationships, and which statistical models to deploy.

Because these choices are often "data-dependent," two equally competent researchers can analyze the same dataset and arrive at opposite conclusions. This issue is particularly acute in typology because, unlike experimental psychology, typologists cannot simply recruit more participants to increase power. There is a finite number of languages in the world, and gathering high-quality data on them is an arduous, slow process. This makes the statistical uncertainty inherent in the "forking paths" both intractable and unavoidable within the bounds of observational data alone.

A Proposed Solution: Methodological Triangulation

Smith proposes a methodological shift toward triangulation—using multiple, independent methods to address the same research question. He suggests a two-step pipeline for the field:

  1. Observational Typology: Large-scale statistical analyses are used to identify potential correlations (e.g., the relationship between community size and grammatical complexity).
  2. Experimental Testing: These correlations are treated as hypotheses to be tested in a laboratory setting using "artificial language learning" paradigms.

By controlling the learning environment, researchers can isolate specific cognitive or interactional pressures. This allows the field to move beyond the "what" (the observed pattern) to the "why" (the causal mechanism).

Key Insights: Typology as the "What," Experiments as the "Why"

This joint approach offers several advantages for the robustness of linguistic theory:

  • Analytic Flexibility as a Feature: The fact that different analyses yield different results is not necessarily a sign of bad science; it reflects the inherent complexity of language. Experiments provide a different kind of data that can help adjudicate between these conflicting statistical interpretations.
  • Reducing Pressure on Typologists: If typology is viewed as part of a joint endeavor with experimentalists, the pressure to identify "the one true universal" with absolute certainty from observational data is lowered. Typology provides the map, while experiments provide the engine.
  • Productive Mismatches: Some of the most valuable insights occur when lab results fail to match typological data. For instance, Smith notes that while "unidirectionality" is a cornerstone of grammaticalization theory, some experiments have failed to find a corresponding cognitive bias. These mismatches force researchers to re-evaluate their theories and look for external factors, such as social or historical pressures, that might be driving the observed patterns.

Evidence from the Field

The paper highlights successful applications of this integrated approach. For example, research into word order in noun phrases (referencing Greenberg’s 1963 universals) has been bolstered by experimental work (e.g., Culbertson et al.). These studies show that the most common word orders found globally are also the easiest for humans to learn in a lab, suggesting a direct link between cognitive biases and typological distributions.

Similarly, Smith’s own work (2024) utilized artificial languages to investigate morphological complexity. By simulating "imperfect learning" by non-native speakers, the research demonstrated that the simplified morphology often observed in large, heterogeneous language communities can indeed be caused by specific social pressures during transmission.

Limitations and Future Directions

Despite the promise of triangulation, Smith acknowledges the "unpalatable compromise" required: to run a controlled experiment, one must strip away the full complexity of natural language. Many typologists find these simplified laboratory models difficult to reconcile with the richness of the languages they study. Furthermore, the community must still reach a consensus on which experimental methods constitute valid tests for specific typological hypotheses.

Ultimately, Smith calls for a move toward more "reanalysis-based accounts." When experiments fail to support long-held typological assumptions, it should be viewed as an opportunity to refine our models. By embracing both the breadth of typology and the precision of the laboratory, linguistics can build a more robust, causally-grounded understanding of how language evolves.

Summary updated

Abstract

Article An experimentalist’s perspective on replicability in typology: commentary on “Replication and methodological robustness in quantitative typology” by Becker and Guzmán Naranjo was published on July 30, 2025 in the journal Linguistic Typology (volume 0, issue 0).
BibTeX
@inproceedings{smith2025experimentalists,
  author = {Kenny Smith},
  title = {{An experimentalist’s perspective on replicability in typology: commentary on “Replication and methodological robustness in quantitative typology” by Becker and Guzmán Naranjo}},
  booktitle = {Linguistic},
  year = {2025},
  doi = {10.1515/lingty-2025-0039},
}
Figure