Beyond Metadata: What Paper Authors Say About Corpora They Use

Nikolay Kolyada, Martin Potthast, Benno SteinACL 2021

TL;DR

This paper tackles the challenge of finding suitable datasets for research by going beyond traditional metadata. The authors introduce a new corpus, Webis-Dataset-Reviews-21, derived from scientific publications, to capture what researchers actually say about datasets they use. This work lays the groundwork for richer dataset search experiences by analyzing these "dataset reviews" to understand user experience and identify key aspects of dataset utility.

What Problem Does This Paper Address?

The explosion of data sharing in science has led to a growing need for effective dataset search tools. While services like Google Dataset Search are a step forward, they primarily rely on metadata provided by dataset creators. This metadata can be inconsistent and doesn't always capture the crucial "user experience" – how well a dataset actually works for a specific research task. Unlike commercial products, there's no established platform for users to share their experiences with datasets. This paper addresses this gap by exploring how to extract valuable insights about dataset usability from the scientific literature itself, where authors often discuss their experiences in detail.

Methods

The authors' approach involves two main steps:

  1. Dataset Metadata Compilation: They first compiled a comprehensive list of NLP datasets by crawling authoritative sources like the ELRA and LDC catalogs. This involved cleaning, normalizing, and merging metadata from over 13,000 entries into a unified schema.
  2. Dataset Review Mining: The core of their work is extracting "dataset reviews" from scientific publications. They processed 57,606 papers from the ACL Anthology, identifying sentences that mention a dataset from their compiled catalog. To capture discussions beyond direct mentions, they employed a co-reference resolution model to link related sentences. They then developed a taxonomy to categorize these mentions into "reuse," "description," and "reference," with "reuse" mentions often indicating a personal experience and thus a potential review.

Datasets

The primary dataset introduced in this paper is the Webis-Dataset-Reviews-21 corpus. This corpus consists of:

  • A unified metadata catalog of 13,372 NLP datasets compiled from sources like ELRA, LDC, and others.
  • Extracted 466,567 mentions of these datasets from 57,606 papers in the ACL Anthology.

The authors also mention using a sample of 1,000 dataset mentions for manual review to derive their taxonomy.

Key Insights and Ideas

  • Beyond Metadata is Crucial: The paper strongly argues that relying solely on metadata for dataset search is insufficient. User experience and practical usability, often discussed in publications, are vital missing pieces of information.
  • Scientific Papers as a Rich Source: Scientific publications, particularly in fields like NLP, contain a wealth of implicit and explicit information about dataset usage, suitability, and limitations. Authors often provide detailed testimonials and criticisms that go beyond what's available in formal metadata.
  • "Reuse" as a Proxy for Review: The authors found that mentions of dataset "reuse" are a strong indicator of a dataset review, as they often describe how the dataset was applied in practice and what the experience was like.
  • NLP Datasets Have Distinct Naming: Unlike some other fields where datasets might have descriptive names, NLP datasets often have proper names or catchy acronyms (e.g., "WordNet"), which aids in their extraction from text.
  • Dataset Popularity Skew: A small number of highly popular datasets (like WordNet and Wikipedia) receive a disproportionately large number of mentions, while the majority of datasets are mentioned infrequently.

Key Results

  • The paper successfully compiled a unified metadata catalog of 13,372 NLP datasets.
  • They extracted 466,567 mentions of these datasets from 57,606 ACL Anthology papers.
  • Their analysis of a sample of mentions revealed a taxonomy of reuse (58%), description (20%), and reference (6%).
  • Crucially, 63% of reuse mentions were identified as sharing a "personal experience," suggesting that approximately 150,000 dataset mentions in the corpus could be considered reviews.
  • The top 20 most mentioned datasets in the ACL Anthology are listed, highlighting the dominance of a few key resources.

Limitations and Future Work

  • Recall of Mentions: The authors acknowledge that their current string-matching approach for extracting mentions has limitations and that future work could explore more sophisticated heuristic or citation-based methods to improve recall. They also note that initial attempts with heuristics introduced too much noise.
  • "Long Tail" Datasets: Identifying datasets in the "long tail" (those mentioned infrequently) will require more tailored named entity recognition approaches.
  • Expanding to Other Fields: The current work focuses on NLP. Future directions include expanding this analysis to other research fields to build a more comprehensive understanding of dataset usage across disciplines.
  • Improving Extraction: Further improvements in extracting mentions, potentially through "few-shot" domain-specific named entity recognition, are suggested.
  • Informing Retrieval Models: The ultimate goal is to use these extracted reviews to inform dataset search engines, enabling them to display richer information and improve retrieval models.

Summary updated

BibTeX
@inproceedings{kolyada2021beyond,
  author = {Nikolay Kolyada and Martin Potthast and Benno Stein},
  title = {{Beyond Metadata: What Paper Authors Say About Corpora They Use}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure