LIORI at SemEval-2021 Task 8: Ask Transformer for measurements

Adis Davletov, Denis Gordeev, Nikolay Arefyev, Emil DavletovACL 2021

TL;DR

This paper presents the winning approach for SemEval-2021 Task 8 (MeasEval), which focuses on extracting measurements and their associated information from text. The authors frame this complex task as a multi-task question-answering problem, leveraging fine-tuned Transformer models like LUKE and RoBERTa to achieve state-of-the-art results.

What Problem Does This Paper Address?

Extracting structured information about measurements from unstructured text is a challenging natural language processing (NLP) task. SemEval-2021 Task 8, MeasEval, aimed to address this by breaking it down into five subtasks: extracting quantity spans (e.g., "355 ml"), classifying units of measurement, identifying measured entities and properties, extracting qualifiers for validation, and finally, extracting relations between all these components. Existing Named Entity Recognition (NER) and relation extraction models often lack the granularity needed for this specific domain, necessitating a tailored approach.

Methods

The core of the LIORI system is a multi-task learning framework built upon Transformer architectures. They fine-tuned two primary models:

  • LUKE: Used for extracting the initial "Quantity" (Q) spans. LUKE is notable for its entity-aware self-attention mechanism, which was adapted for this NER task.
  • RoBERTa: Fine-tuned to extract all other information related to the identified quantities, including units of measurement (UoM), measured entities (ME), measured properties (MP), and qualifiers (Qlfr).

The system treats each subtask as a question-answering (QA) problem. For instance, to extract a quantity, the model might be prompted with a question like "What is the quantity?". They also employed a novel approach of using learnable scalar weights to balance the contribution of each subtask to the overall loss during multi-task training. Data augmentation, including concatenating sentences and using different weighting schemes for subtasks, was also explored to improve performance.

Datasets

The primary dataset used was the SemEval-2021 Task 8: MeasEval dataset. This dataset was divided into training, trial, and evaluation sets, containing annotated entities across five classes: Quantity (Q), Measured Entity (ME), Measured Property (MP), Unit of Measurement (UoM), and Qualifier (Qlfr). The data was distributed across various domains, including Agriculture, Astronomy, Biology, Chemistry, Computer Science, Earth Science, Engineering, Materials Science, Mathematics, and Medicine.

Key Insights and Ideas

  • Framing as Question Answering: Treating measurement extraction as a question-answering task proved highly effective, allowing the use of powerful pre-trained language models in a flexible way.
  • Multi-Task Learning with Learnable Weights: The ability to dynamically weight subtasks within a multi-task learning framework was crucial for optimizing performance across the diverse set of subtasks.
  • Leveraging LUKE for Span Extraction: The entity-aware capabilities of LUKE were particularly beneficial for the initial, critical step of accurately identifying quantity spans.
  • Data Augmentation Strategies: Techniques like sentence concatenation and careful selection of weighting schemes significantly boosted performance, highlighting the importance of robust training data.

Key Results

The LIORI system achieved first place in the SemEval-2021 Task 8: MeasEval competition. While specific F1 scores for each subtask are detailed in the tables, the overall success indicates strong performance across all five subtasks. The paper highlights that using an "equal" weighting scheme for subtasks, combined with data augmentation, yielded the best results on both development and test sets. They also found that fine-tuning larger models like RoBERTa-large did not always outperform smaller counterparts, suggesting that careful hyperparameter tuning and weighting strategies are paramount.

Limitations and Future Work

The authors acknowledge that while their system achieved state-of-the-art results, there might still be duplicate predictions for the same quantity if different extracted values exist. They also note that their submitted results were higher than those without test-time data augmentation, implying that such augmentation is beneficial. Future work could explore more sophisticated methods for handling duplicate predictions and further investigate the impact of different model architectures and training strategies on measurement extraction tasks.

Summary updated

BibTeX
@inproceedings{davletov2021liori,
  author = {Adis Davletov and Denis Gordeev and Nikolay Arefyev and Emil Davletov},
  title = {{LIORI at SemEval-2021 Task 8: Ask Transformer for measurements}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure