Introduction
For years, the Natural Language Processing (NLP) community has focused on debunking "hype" surrounding artificial intelligence. However, this paper argues that the field has overcorrected into a state of underclaiming—systematically downplaying the actual capabilities of state-of-the-art (SOTA) models. This trend of scholarly pessimism is not merely a matter of scientific modesty; it poses significant risks to researcher credibility and our collective ability to prepare for transformative AI developments.
The Four Archetypes of Underclaiming
Samuel Bowman identifies four primary ways in which researchers inadvertently underclaim model capabilities:
- Reliance on Outdated Baselines: Researchers often cite the failures of older models (e.g., from 2017) to imply that current SOTA models share the same fundamental flaws, ignoring the rapid pace of improvement.
- The Analysis Lag: There is often a 2-3 year delay between the release of a new model and the publication of deep analysis papers. Consequently, many critiques focus on models like BERT, even after newer architectures like DeBERTa have resolved the specific weaknesses being analyzed.
- Misapplication of Philosophical Limits: Arguments regarding the theoretical limits of learning from "form" alone (e.g., the "Octopus" thought experiment) are sometimes incorrectly used to predict that models will fail at practical, empirical tasks they are already performing.
- Misinterpretation of Adversarial Datasets: Low scores on datasets specifically designed to be "tricks" (like ANLI) are often reported as absolute measures of general capability, rather than as stress tests for specific edge cases.
Empirical Evidence of Progress
To counter the narrative of stagnation, the paper highlights quantitative leaps in model performance. A key example is the progress made on the Stanford Question Answering Dataset (SQuAD) and its adversarial variants (AS and AOS).
As shown in the table above, error rates on adversarial sets were significantly reduced in just two years. Between the ReasoNet ensemble (2017) and XLNet-Base (2019), accuracy on the AS metric jumped from 39 to 69, effectively cutting the error rate nearly in half (from 61% to 31%).
The Risks of Scholarly Pessimism
Underclaiming creates a "credibility trap." If researchers insist that a system "cannot understand language" while industry stakeholders observe that same system successfully automating complex linguistic tasks, the research community loses its influence. This disconnect makes it harder to advocate for safety measures or address subtle harms like algorithmic bias.
Furthermore, underclaiming leaves the community unprepared for rapid scaling. If we convince ourselves that certain capabilities are impossible, we risk being blindsided by the arrival of transformative, and potentially catastrophic, AI risks.
Conclusion and The "Bender Rule"
To improve reporting, Bowman proposes a new heuristic: when reporting a model failure, researchers must specify the model type, its relation to current SOTA, and whether the test was an adversarial "trick." Moving forward, the NLP community must balance its skepticism with empirical forecasting to ensure we are ready for the reality of increasingly capable AI systems.
Summary updated