Targeting the Benchmark: On Methodology in Current Natural Language Processing Research

David SchlangenACL 2021

TL;DR

This paper argues that the current benchmark-driven methodology in NLP, while effective at driving incremental progress on specific tasks, often obscures the bigger picture of understanding language. It calls for a more explicit connection between tasks, the underlying linguistic capabilities they aim to test, and a clearer understanding of what constitutes genuine progress in modeling human language competence.

What Problem Does This Paper Address?

The field of Natural Language Processing (NLP) has seen tremendous progress, largely fueled by the creation of benchmarks. These benchmarks typically consist of a specific language task and a dataset designed to evaluate models on that task. The pattern is familiar: a task is introduced, a baseline model is established, and then subsequent research focuses on improving performance on that specific task. However, the paper questions whether this relentless focus on benchmarks truly leads to a deeper understanding of language itself. It highlights a lack of explicit argumentation for why achieving a certain score on a benchmark constitutes progress towards the foundational goals of NLP, such as modeling human language competence. The core problem is the implicit nature of the assumptions connecting tasks, datasets, and the underlying cognitive capabilities we aim to replicate.

Methods

The paper doesn't introduce new algorithms or models. Instead, it offers a methodological critique and proposes a framework for analyzing current NLP research practices. The author dissects the components of a benchmark-driven methodology:

  • Language Task: Defined as a mapping between input and output spaces, where at least one involves natural language. Tasks are often described "intensionally" (e.g., "translation") and "extensionally" via datasets.
  • Dataset: A collection of examples for a task, crucial for training and evaluating models. The paper discusses the importance of datasets truly exemplifying the task and not being susceptible to "exploitable biases."
  • Benchmark: A task specifically designed to evaluate the performance of an algorithm, often with the goal of comparing it to other systems. A key distinction in NLP is that the benchmark often evaluates the learning algorithm itself, rather than a complete system.

The author then analyzes how tasks are evaluated, distinguishing between:

  • Product Tasks: Tasks with direct value to consumers (e.g., translation).
  • Annotation Tasks: Tasks motivated by linguistic theory.
  • Benchmark Tasks: Tasks valued for their ability to test a particular capability and discriminate between learners.

Datasets

The paper does not introduce new datasets. Instead, it discusses the role of datasets in existing benchmarks and highlights the need for datasets that faithfully exemplify the intended task and avoid "exploitable biases." Examples of tasks that have datasets mentioned include:

  • Visual Question Answering (VQA)
  • Natural Language Inference (NLI)
  • Image Description

Key Insights and Ideas

  • The "Benchmark Treadmill": The current methodology can lead to a cycle of incremental improvements on isolated tasks without necessarily advancing our fundamental understanding of language.
  • Implicit Assumptions: The connection between a benchmark task and the underlying linguistic capabilities it's supposed to test is often implicit, making it hard to assess genuine progress.
  • Task vs. Dataset: A model performing well on a dataset doesn't automatically mean it has mastered the intended task or the underlying capability. The dataset itself might contain shortcuts.
  • The Value of "Why": Progress in NLP should be framed not just by what models can do, but why they can do it, and how that relates to human language competence.
  • Beyond Single Tasks: While multi-task learning and pre-training have emerged, the paper suggests a need for a more systematic approach to understanding how capabilities are composed and how tasks can be organized into a curriculum.

Key Results

The paper is primarily a conceptual and methodological analysis, not an empirical study with quantitative results. Therefore, there are no specific accuracy scores or performance metrics to report. The "results" are in the form of insights and arguments about the current state of NLP research methodology.

Limitations and Future Work

The author acknowledges that the discussion is largely "idealised (or normative)" and that in reality, there's often a lack of explicitness regarding the connections between tasks and capabilities.

Future work suggested includes:

  • Explicitly connecting tasks to capabilities: Developing clearer theories about how language competence decomposes into testable capabilities.
  • Assembling a task curriculum: Organizing tasks in a complexity and inclusion hierarchy to guide research towards a developmental trajectory of language understanding.
  • Reconnecting with linguistics and cognitive psychology: Drawing on these fields to better understand and specify linguistic capabilities.
  • Focusing on transferability: Investigating how knowledge gained from modeling one task can be effectively transferred to others.

Summary updated

BibTeX
@inproceedings{schlangen2021targeting,
  author = {David Schlangen},
  title = {{Targeting the Benchmark: On Methodology in Current Natural Language Processing Research}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}

Loading related content...

Figure