Square One Bias in NLP: Towards a Multi-Dimensional Exploration of the Research Manifold

Sebastian Ruder, Ivan Vulic, Anders Søgaard

Introduction

In the evolution of Natural Language Processing (NLP), certain experimental paradigms have become so entrenched that they define the "standard" way of doing science. This prototypical setup—training a standard architecture on labeled English data and optimizing solely for accuracy—is what researchers call the Square One Bias. While this focus helped the field mature, it has created a "one-dimensional" rut where innovation typically moves along only a single axis at a time, leaving the complex interactions between fairness, efficiency, and multilinguality largely unexplored.

The One-Dimensional Trap

The "research manifold" represents the space of all possible NLP discoveries across multiple dimensions. Ideally, research would explore the interior of this space, looking at how different goals interact. However, a manual classification of recent literature reveals that most papers that depart from the "Square One" prototype do so in a strictly linear fashion.

The NLP research manifold, showing how papers cluster at 'Square One' or along single dimensions.

As seen in the visualization above, papers tend to cluster either at the origin (Square One) or along the primary axes. A paper might explore fairness, but it will do so only in English. Another might explore multilinguality but ignore computational efficiency. This one-dimensionality is often a survival strategy for researchers; the community rewards concise, incremental progress that is easy for reviewers to digest. Unfortunately, this means we miss critical "blind spots." For example, weight pruning (an efficiency technique) has been shown to disproportionately hurt performance on minority demographics—a failure mode that is invisible if you only study efficiency in a vacuum.

Quantitative Evidence from ACL 2021

To quantify this bias, the authors annotated 461 oral papers from ACL 2021 across four key dimensions: multilinguality, fairness/bias, efficiency, and interpretability. The results paint a stark picture of a field tethered to its origins.

Distribution of research dimensions across ACL 2021 oral papers by area.

The meta-analysis revealed that approximately 70% of papers evaluate only on English, and nearly 40% use only Accuracy or F1 as their primary metric. Most tellingly, only 6.1% of papers made a contribution along two or more dimensions simultaneously. Even in specialized tracks, the silos remain: as shown in the table, papers in the "Ethics in NLP" track were 100% focused on fairness but 0% on multilinguality or efficiency.

Systemic Bottlenecks

Why is the field so resistant to multi-dimensional research? The authors identify several systemic barriers:

  1. Education: Introductory NLP courses often frame the field through the lens of English-centric tasks like sentiment analysis. Survey data shows that 75% of teachers start with these "Square One" examples, imprinting them as the default scientific method for new researchers.
  2. Organizational Silos: Conference tracks are organized by topic (e.g., "Fairness" vs. "Machine Translation"). A paper that explores the intersection of both might be seen as "not deep enough" for either track, leading to rejection in favor of narrower, more specialized work.
  3. The "State-of-the-Art" (SOTA) Race: The pressure to beat a single accuracy number on a leaderboard discourages researchers from considering trade-offs, such as a model that is slightly less accurate but significantly more efficient or fair.

Recommendations for a Multi-Dimensional Future

To break out of the Square One Bias, the authors propose several practical shifts:

  • Multi-Dimensional Leaderboards: We should move toward "Pareto frontiers" that visualize the trade-offs between speed, fairness, and language coverage rather than a single-column ranking.
  • Reviewer Reform: Conferences should incentivize "inter-areal" reviews, ensuring that papers exploring intersections are evaluated by people who value the breadth of the contribution.
  • Curriculum Updates: Educators should introduce multilinguality and ethics as core components of "NLP 101" rather than optional, advanced topics.

By acknowledging the Square One Bias, the NLP community can begin to explore the full manifold of research, moving beyond simple accuracy to build systems that are robust, equitable, and efficient across the world's languages.

Summary updated

Abstract

The prototypical NLP experiment trains a standard architecture on labeled English data and optimizes for accuracy, without accounting for other dimensions such as fairness, interpretability, or computational efficiency. We show through a manual classification of recent NLP research papers that this is indeed the case and refer to it as the square one experimental setup. We observe that NLP research often goes beyond the square one setup, e.g, focusing not only on accuracy, but also on fairness or interpretability, but typically only along a single dimension. Most work targeting multilinguality, for example, considers only accuracy; most work on fairness or interpretability considers only English; and so on. Such one-dimensionality of most research means we are only exploring a fraction of the NLP research search space. We provide historical and recent examples of how the square one bias has led researchers to draw false conclusions or make unwise choices, point to promising yet unexplored directions on the research manifold, and make practical recommendations to enable more multi-dimensional research. We open-source the results of our annotations to enable further analysis.
BibTeX
@inproceedings{ruder2022square,
  author = {Sebastian Ruder and Ivan Vulic and Anders Søgaard},
  title = {{Square One Bias in NLP: Towards a Multi-Dimensional Exploration of the Research Manifold}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
  url = {https://openreview.net/forum?id=57QiEoKPkL},
}
Figure