Benchmarking: Past, Present and Future

Kenneth Church, Mark Liberman, Valia KordoniACL 2021

TL;DR

This paper is a collection of abstracts from an invited workshop on the past, present, and future of benchmarking in Natural Language Processing (NLP). It highlights the evolution of benchmarking from early, top-down approaches to more dynamic, bottom-up methods, and discusses critical challenges like reliability, bias, and the need for more realistic evaluation settings. The overarching theme is the urgent need to rethink and improve NLP evaluation to truly measure progress and drive meaningful advancements.

What Problem Does This Paper Address?

The field of Natural Language Processing (NLP) has seen rapid advancements, particularly with the rise of deep learning. However, the methods used to evaluate these advancements, known as benchmarking, are struggling to keep pace. Current benchmarks often become saturated quickly, are susceptible to "overfitting" where models learn to game the benchmark rather than truly understand, and can be influenced by annotator bias or exploitable artifacts in the data. Furthermore, many benchmarks don't accurately reflect real-world scenarios, leading to a gap between performance on benchmarks and actual utility. This collection of abstracts addresses the critical need to re-evaluate and evolve NLP benchmarking practices to ensure they are reliable, robust, and truly measure progress towards genuine language understanding.

Methods

This paper doesn't present a single, unified methodology. Instead, it's a compilation of perspectives from various researchers, each discussing their approach to benchmarking or the challenges they've encountered. Key themes and proposed methods include:

  • Rethinking Dataset Design: Moving beyond static, easily gamed datasets to those that better reflect real-world complexity, including common sense, social context, and encyclopedic knowledge.
  • Adversarial and Out-of-Distribution Testing: Creating test sets that are intentionally difficult or different from training data to assess model robustness.
  • Interactive Evaluation: Developing methodologies for evaluating NLP systems in interactive settings, potentially with human-in-the-loop assistance, to allow for more personalized and exploratory behavior.
  • Explicit Representations: Exploring the use of explicit representations for textual information structure, moving beyond purely end-to-end models.
  • Common Task Method: A historical approach, championed by DARPA, that involves defining specific tasks with common training and test corpora to foster incremental progress and allow for international collaboration.
  • Industry-Specific Benchmarks: Developing datasets and benchmarks that directly address the challenges and requirements of real-world industrial applications, often focusing on robustness and adaptability across multiple domains.
  • Focus on Reliability and Bias: Addressing issues of data sampling bias, human annotation bias, and developing more suitable evaluation metrics.

Datasets

The paper mentions several existing datasets and challenges, and implicitly discusses the need for new ones:

  • Pitt Corpus in DementiaBank: Used for dementia detection from speech.
  • ADReSS challenge: A challenge for INTERSPEECH2020 related to dementia detection.
  • GLUE and SuperGLUE: Widely used benchmark competitions for natural language understanding.
  • LUGE: An open-source project for Chinese NLP benchmarks, aiming to evaluate robustness and adaptability.
  • DIHARD evaluation series: Focuses on speaker diarization.
  • TREC (Text REtrieval Conference): A long-standing series of community evaluations for information retrieval.

The paper also highlights the need for datasets that are more representative of real-world scenarios, include longitudinal data, and cover diverse language tasks and interaction types, particularly beyond English.

Key Insights and Ideas

  • Benchmarking is Evolving: The shift from top-down, government-led initiatives to more bottom-up, community-driven efforts (like "papers with code") is a significant trend.
  • "Broken" NLU Evaluation: Current NLU benchmarks are often unreliable and biased, allowing systems to score highly without truly understanding. Adversarial testing, while useful, doesn't fully solve the underlying issues.
  • The Importance of Realism: Benchmarks need to move beyond "comfort zones" and reflect realistic settings, including limited training data (few-shot, transfer learning) and diverse data compositions.
  • Beyond Accuracy: Simply comparing accuracy scores is insufficient. Nuanced studies are needed to interpret benchmark results, considering factors like computational resources and the dynamic nature of language.
  • The "Common Task Method" Success: DARPA's approach to creating common tasks has been instrumental in driving progress in areas like speech recognition and machine translation.
  • Industry vs. Academia: Datasets created by academia might not always align with the distributions and challenges faced in real-world industrial applications, necessitating industry-led efforts.
  • The Need for Robustness and Adaptability: For NLP models to succeed in industrial settings, they must be robust to variations and adaptable across multiple tasks and domains.

Key Results

This paper is a collection of abstracts and does not present a single set of quantitative results. Instead, it discusses the state of benchmarking and the challenges that need to be addressed. For example, it notes that the DIHARD challenge has seen significant improvements in error rates over a few years, but robust diarization remains elusive. The paper implies that current benchmarks are not yet achieving "human parity" in many areas of natural language understanding.

Limitations and Future Work

  • Limitations of Current Benchmarks: The paper repeatedly points out the limitations of existing benchmarks, including saturation, susceptibility to overfitting, annotator artifacts, unclear metrics, and a lack of real-world relevance.
  • Need for Dynamic Benchmarks: The static nature of many benchmarks is a problem as language and world knowledge evolve.
  • Bias in Data and Annotation: Addressing social bias in datasets and annotation processes is a critical area for future work.
  • Multilinguality: The paper highlights that as NLP moves beyond English, multilinguality introduces significant challenges and opportunities for benchmarking.
  • Future Directions: The authors suggest a need for progress in dataset design, reliability of annotation, dataset size, and methods for handling social bias. They also emphasize the importance of interactive evaluation, explicit representations, and benchmarks that better simulate real-life scenarios. The development of new, more relevant datasets and evaluation methodologies is a clear call to action.

Summary updated

BibTeX
@inproceedings{church2021benchmarking,
  author = {Kenneth Church and Mark Liberman and Valia Kordoni},
  title = {{Benchmarking: Past, Present and Future}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure