When Combating Hype, Proceed with Caution

Samuel R. Bowman

Abstract

Researchers in NLP often frame and discuss research results in ways that serve to deemphasize the field's successes, often in response to the field's widespread hype. Though well-meaning, this has yielded many misleading or false claims about the limits of our best technology. This is a problem, and it may be more serious than it looks: It harms our credibility in ways that can make it harder to mitigate present-day harms, like those involving biased systems for content moderation or resume screening. It also limits our ability to prepare for the potentially enormous impacts of more distant future advances. This paper urges researchers to be careful about these claims and suggests some research directions and communication strategies that will make it easier to avoid or rebut them.

Introduction

For years, the Natural Language Processing (NLP) community has been locked in a battle against "hype"—the tendency for industry PR and the media to overstate the capabilities of AI systems. In a well-intentioned effort to maintain scientific rigor, researchers have become experts at identifying where models break. However, this pendulum has swung too far toward underclaiming: a systematic tendency to downplay the actual capabilities of state-of-the-art (SOTA) systems. This unjustified pessimism threatens our credibility with stakeholders and leaves society unprepared for the rapid advancement of technology.

The Mechanics of Underclaiming

Underclaiming often manifests through specific rhetorical and methodological patterns. A primary driver is the "Three-Year Lag": the significant time gap between a model's release and the publication of deep analysis papers. By the time we understand why a model like BERT fails, the industry has often moved on to models that do not share those specific weaknesses.

Figure 1: Citation trends for a seminal 2017 critique, showing continued influence and peak citations years after the evaluated models were surpassed.

This lag leads to several problematic research practices:

  1. Citing Dated Models: Characterizing the failures of 2022 models by citing papers from 2017 (such as the Jia and Liang adversarial SQuAD paper), despite error rates having been significantly reduced in the interim.
  2. Theoretical vs. Behavioral Misinterpretation: Theoretical arguments (like those in "Stochastic Parrots") stating that models cannot learn meaning from form alone are often misinterpreted as behavioral predictions that models will fail at reasoning tasks.
  3. Adversarial Over-indexing: Treating 0% accuracy on a test set specifically designed to trick a model as a sign of total incompetence, rather than a measure of the test's success in finding edge cases.

Quantitative Evidence of Progress

The gap between perceived brittleness and actual performance is narrowing faster than many critiques acknowledge. For example, on the Winograd Schema Challenge—a test of commonsense reasoning—BERT performed near chance. However, newer models like DeBERTa have reached 96% accuracy, essentially solving the task.

Similar progress is visible in reading comprehension. While early neural models were easily fooled by distracting sentences in adversarial versions of SQuAD, subsequent architectures rapidly recovered that performance.

Table 1: Performance gains on SQuAD and its adversarial variants (AS, AOS) from 2017 to 2019, showing how newer models like XLNet significantly improved on "impossible" adversarial sets.

The Risks of Strategic Pessimism

Researchers sometimes downplay capabilities to discourage the reckless deployment of biased systems. However, this strategy often backfires. If a developer sees that a system actually works for their specific business case, they will stop trusting researchers who claim the system "doesn't understand language."

Furthermore, confusing a lack of "grounded" theoretical understanding with a lack of behavioral performance leads to a false sense of security. If we convince ourselves that models are fundamentally incapable, we will be unprepared to align these powerful systems with human values as they continue to scale.

Conclusion and Recommendations

To combat underclaiming without falling back into hype, the community should adopt a "Rule of Thumb" for reporting failures. Researchers should explicitly state: (i) exactly what kind of model failed, (ii) if it is significantly less capable than the current SOTA, and (iii) if the test was specifically designed to "trick" the model. By prioritizing precision over pessimism, we can better prepare for both the present harms and future risks of AI.

Summary updated

Abstract

Researchers in NLP often frame and discuss research results in ways that serve to deemphasize the field's successes, often in response to the field's widespread hype. Though well-meaning, this has yielded many misleading or false claims about the limits of our best technology. This is a problem, and it may be more serious than it looks: It harms our credibility in ways that can make it harder to mitigate present-day harms, like those involving biased systems for content moderation or resume screening. It also limits our ability to prepare for the potentially enormous impacts of more distant future advances. This paper urges researchers to be careful about these claims and suggests some research directions and communication strategies that will make it easier to avoid or rebut them.
BibTeX
@inproceedings{bowman2022when,
  author = {Samuel R. Bowman},
  title = {{When Combating Hype, Proceed with Caution}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
  url = {https://openreview.net/forum?id=zlT1lL4Fl7w},
}
Figure