Abstract
The prototypical NLP experiment trains a standard architecture on labeled English data and optimizes for accuracy, without accounting for other dimensions such as fairness, interpretability, or computational efficiency. We show through a manual classification of recent NLP research papers that this is indeed the case and refer to it as the "square one" experimental setup. We observe that NLP research often goes beyond the square one setup—focusing not only on accuracy, but also on fairness or interpretability—but typically only along a single dimension. Most work targeting multilinguality, for example, considers only accuracy; most work on fairness or interpretability considers only English. This one-dimensionality means we are only exploring a fraction of the NLP research search space. We provide historical and recent examples of how the square one bias has led to false conclusions, point to promising yet unexplored directions, and make practical recommendations to enable more multi-dimensional research.
Introduction: The "Square One" Prototype
In the evolution of scientific fields, researchers often converge on a "prototype"—a standard experimental configuration that serves as a baseline for progress. In Natural Language Processing (NLP), this prototype typically consists of a standard architecture (such as a Transformer) trained on English data and evaluated using a single performance metric like Accuracy or F1. While this standardization has allowed for rapid benchmarking, it has created what we term the "Square One Bias."
This bias incentivizes researchers to move away from the prototype in only one direction at a time. A paper might investigate model efficiency while remaining strictly within the English language domain, or it might explore fairness while ignoring computational constraints. This "one-dimensional" approach prevents the community from understanding the complex, non-linear interactions between these critical dimensions, effectively leaving the interior of the research manifold unexplored.
Mapping the Field: A Meta-Analysis of NLP Research
To quantify this bias, we conducted a rigorous manual annotation of 461 oral papers from ACL 2021 and 20 papers from EMNLP 2021. We categorized these works across four key dimensions beyond standard accuracy: Multilinguality, Fairness/Bias, Efficiency, and Interpretability.
Our findings reveal a stark reality regarding the current state of the field:
- The 6% Problem: Only 6.1% of the analyzed papers made a contribution along more than one dimension beyond the standard setup.
- The Dominance of Square One: 56.6% of papers remained entirely at "Square One," focusing solely on accuracy within a standard English-centric framework.
- The Gravity of English: Despite significant advocacy for linguistic diversity, nearly 70% of papers evaluated their methods exclusively on English data.
The Dimensions of Departure
When researchers do leave Square One, they tend to follow predictable, isolated paths. Efficiency is the most common departure (17.8%), followed by multilinguality (13.9%). In contrast, fairness remains the rarest focus, addressed by only 6.3% of oral papers.
Crucially, these dimensions are rarely combined. The most frequent multi-dimensional pairing—multilinguality and efficiency—appeared in only 17 papers. Even more telling, only a single paper out of the 461 annotated considered three dimensions (Fairness, Efficiency, and Multilinguality) simultaneously. This suggests that the research space is being explored only along its edges, failing to account for how these goals might conflict or synergize.
Why Does the Bias Persist?
The Square One Bias is reinforced by the field's educational, historical, and organizational structures:
1. Educational Reinforcement
A survey of 71 NLP instructors globally, alongside an analysis of five major textbooks spanning 20 years, shows that "first experiments" taught to students are almost exclusively English-centric (e.g., sentiment analysis or topic modeling). This pedagogical default biases the next generation of researchers to view English and accuracy as the "natural" starting point for all research.
2. Historical Precedents
An analysis of "Test-of-Time" award winners from the last two years shows that 7 out of 8 recipients performed evaluations exclusively on English data. This suggests that the community’s highest honors are still frequently awarded to research that adheres to the Square One prototype, signaling to researchers that multi-dimensionality is not required for high-impact work.
3. Organizational Silos
Conference structures often divide papers into tracks like "Fairness" or "Efficiency." While intended to organize expertise, these silos can inadvertently discourage multi-dimensional work. A reviewer focused on efficiency may not fully appreciate the magnitude of a fairness contribution, leading to lower scores for papers that attempt to bridge these gaps.
The Risks of One-Dimensionality
The danger of this bias is that dimensions are not independent. For example, research has shown that "weight pruning"—a common efficiency technique—can have disparate impacts on different demographic groups. If we only study efficiency in isolation, we risk developing models that are fast but fundamentally unfair. By failing to explore the intersections of the research manifold, we may draw false conclusions about the generalizability and safety of our models.
Recommendations for a Multi-Dimensional Future
To move beyond the edges of the research manifold, we propose several structural changes:
- Multi-Dimensional Checklists: Conferences should adopt checklists that prompt researchers to consider dimensions like fairness, efficiency, and multilinguality, even if they are not the primary focus of the paper.
- Reviewer Reform: Program committees should assign reviewers with complementary expertise (e.g., pairing an interpretability expert with a multilinguality expert) to evaluate multi-dimensional submissions.
- Curriculum Updates: NLP 101 courses should integrate multilinguality and ethics as core components of the "first experiment" rather than treating them as advanced or optional topics.
By acknowledging and actively countering the Square One Bias, the NLP community can begin to explore the full complexity of the research manifold, leading to more robust, equitable, and efficient language technologies.
The results of our annotations are open-sourced to enable further analysis at https://github.com/google-research/url-nlp.
Summary updated