SemEval-2021 Task 8: MeasEval – Extracting Counts and Measurements and their Related Contexts

Corey Harper, Jessica Cox, Curt Kohler, Antony Scerri, Ron Daniel Jr., Paul GrothACL 2021

TL;DR

This paper introduces MeasEval, a new SemEval task focused on automatically extracting numerical measurements and their associated context from scientific literature. This is crucial for building knowledge graphs and enabling more sophisticated scientific data analysis, addressing the challenge of ambiguous and inconsistent scientific writing.

What Problem Does This Paper Address?

Extracting precise numerical information from scientific texts is surprisingly difficult. While identifying a number like "17mg" is easy, understanding what it refers to (e.g., medicine dosage vs. concrete additive) is a significant challenge. This ambiguity stems from the inconsistent ways scientists write, making it hard to automatically build structured knowledge bases or perform large-scale data analysis. Existing methods often struggle to capture the full context, such as the measured entity, property, and any qualifying conditions, which are essential for making these measurements useful.

Methods

MeasEval frames the problem as an entity recognition and semantic relation extraction task. It breaks down the extraction into five sub-tasks:

  • Quantity Extraction: Identifying spans representing counts or measurements (e.g., "12 kg").
  • Unit Extraction: Identifying the unit associated with a quantity (e.g., "kg").
  • Modifier Extraction: Identifying value modifiers like "approximate," "range," or "mean."
  • Entity and Property Extraction: Identifying the "what" (Measured Entity, e.g., "bed inventory") and the "of what" (Measured Property, e.g., "concentration") that the quantity describes.
  • Qualifier Extraction: Identifying contextual information that helps validate or understand the measurement (e.g., "after incubation," "at 25°C").

The task also involves extracting relationships between these components (e.g., HasQuantity, HasProperty, Qualifies). The annotation model is designed to be flexible, with relationships being optional.

Datasets

The MeasEval dataset is built upon 110 CC-BY licensed articles previously made available by Elsevier Labs. These articles were split into a training set of 1164 quantities and an evaluation set of 499 quantities. A subset of 248 quantities was used for inter-annotator agreement analysis. The data is provided in both tab-separated value (.tsv) and BRAT annotation formats.

Key Insights and Ideas

  • Context is King: Simply extracting numbers isn't enough; the real value lies in understanding their context. MeasEval highlights the necessity of linking quantities to their measured entities, properties, and qualifying conditions.
  • Qualifier Challenge: The Qualifier span and its associated relationship proved to be the most difficult aspect of the task, indicating that capturing nuanced contextual information is a significant hurdle.
  • Pipeline vs. Joint Models: While many top systems used pipeline approaches (e.g., first extracting quantities, then other elements), some explored joint inference models, suggesting ongoing research into the most effective architectures.
  • BERT Dominance: The BERT family of pre-trained models (BERT, SciBERT, BioBERT, RoBERTa) was the dominant architecture among participating systems, underscoring its effectiveness in natural language understanding tasks.

Key Results

The MeasEval task saw over 75 submissions from 25 participants. The top-performing systems achieved an overall F1 (Overlap) score of around 0.519. Quantity and Unit extraction were generally the easiest subtasks, with high average scores. Modifiers also performed relatively well. However, Measured Entity, Measured Property, and especially Qualifier extraction, along with their associated relations, presented greater challenges, with significantly lower scores. The baseline model achieved an overall F1 of 0.239.

Limitations and Future Work

The authors acknowledge that the relatively small training data size and the inherent complexity of inter-relationships between extracted components made the task challenging. They also note that the annotation process itself showed variation in interpretation among humans, particularly for Qualifiers, suggesting that different interpretations might be useful in different downstream applications. Future work could focus on further analyzing areas of error and disagreement, investigating entity linking to measurement ontologies, and exploring how the extracted information can be integrated into scientific knowledge graphs and downstream applications. The impact of duplicates in submissions was also analyzed, leading to a post-processing step to remove them before final evaluation.

Summary updated

BibTeX
@inproceedings{harper2021semeval,
  author = {Corey Harper and Jessica Cox and Curt Kohler and Antony Scerri and Ron Daniel Jr. and Paul Groth},
  title = {{SemEval-2021 Task 8: MeasEval – Extracting Counts and Measurements and their Related Contexts}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure