MuLMS-AZ: An Argumentative Zoning Dataset for the Materials Science Domain

Timo Pierre Schrader, Teresa Bürkle, Sophie Henning, Sherry Tan, Matteo Finco, Stefan Grünewald, Maira Indrikova, Felix Hildebrand, Annemarie FriedrichACL 2023

TL;DR

Researchers have created MuLMS-AZ, the first large-scale dataset for Argumentative Zoning (AZ) specifically tailored to materials science research. This dataset, annotated by domain experts, aims to improve the understanding and processing of scientific texts by classifying sentences based on their rhetorical roles, a task crucial for applications like summarization and information extraction.

What Problem Does This Paper Address?

Academic writing follows a structured approach to present arguments and convince readers. Argumentative Zoning (AZ) is a Natural Language Processing (NLP) task that aims to identify the rhetorical function of sentences within a document (e.g., stating a motivation, a result, or background information). While AZ datasets exist for fields like computational linguistics and computer graphics, there's a lack of such resources for materials science. This gap makes it difficult to apply existing AZ models directly to materials science literature, which has its own unique rhetorical conventions and technical jargon. The authors address this by creating a new, domain-specific AZ dataset and exploring how well existing models and cross-domain transfer learning perform on it.

Methods

The core of this work involves creating and annotating a new dataset, MuLMS-AZ. The annotation scheme was developed in collaboration with materials science domain experts, resulting in a hierarchical, multi-label system that includes labels specific to the field, such as "PREPARATION" and "CHARACTERIZATION." To handle the class imbalance inherent in such datasets, the authors employed a multi-label random oversampling (ML-ROS) algorithm. For modeling, they used BERT-based transformer architectures, specifically testing domain-specific variants like SciBERT and MatSciBERT, and explored multi-task learning with existing AZ datasets.

Datasets

The primary contribution is the MuLMS-AZ dataset, comprising 50 manually annotated materials science research articles, totaling over 10,000 sentences. This dataset is unique in its focus on the materials science domain and its hierarchical, multi-label annotation scheme. The authors also used existing AZ datasets for multi-task learning experiments, including:

  • AZ-CL corpus: Computational linguistics papers.
  • ART corpus: Physical chemistry and biochemistry papers.
  • DRI corpus: Computer graphics papers.
  • PubMed corpus: Biomedical abstracts.

Key Insights and Ideas

  • Domain-Specific Annotations are Crucial: The creation of MuLMS-AZ highlights that standard AZ labels might not fully capture the nuances of scientific discourse in specialized fields like materials science. Labels like "PREPARATION" and "CHARACTERIZATION" are essential for understanding materials science research.
  • Domain-Specific Embeddings Matter: Using pre-trained language models specifically trained on scientific text (like SciBERT and MatSciBERT) significantly outperforms general-purpose models (like BERT) for AZ tasks in this domain.
  • Transfer Learning Has Limits: While multi-task learning with other AZ datasets can provide some benefits, the effectiveness of transferring knowledge is not uniform. The AZ-CL corpus showed more promise than the ART corpus, suggesting that even with domain differences, some rhetorical structures might be more transferable than others.
  • Class Imbalance is a Real Challenge: The dataset exhibits significant class imbalance, and techniques like ML-ROS are necessary to achieve good performance on minority labels.

Key Results

  • Strong Baseline Performance: The MuLMS-AZ dataset allows for the training of models that achieve high classification performance, with domain-specific BERT models (SciBERT) outperforming general BERT.
  • ML-ROS Benefits: Using ML-ROS for handling class imbalance led to minor but consistent increases in performance across most labels, with notable improvements for minority classes like MOTIVATION.
  • Multi-Task Learning Gains: Multi-task learning with the AZ-CL dataset provided a modest boost in both micro-F1 and macro-F1 scores, particularly for labels like CHARACTERIZATION and METADATA.
  • Inter-Annotator Agreement: The dataset demonstrates substantial inter-annotator agreement (Krippendorff's alpha values ranging from 0.39 to 0.89), indicating the reliability of the annotations.

Limitations and Future Work

  • Single Annotator for Core Dataset: The primary annotation was done by a single graduate student. While an agreement study was conducted with another expert, the core dataset relies on one annotator.
  • Limited Scope of Experiments: The study focused on BERT-based transformers and did not explore other model classes like document-level models or CRF-based approaches, which could be beneficial for capturing inter-sentence dependencies.
  • Limited Transfer Learning Exploration: Only one method for handling class imbalance (ML-ROS) was tested, and further exploration of other techniques or data augmentation methods is suggested.
  • Future Directions: The authors suggest exploring cross-domain AZ, more advanced methods for class imbalance, and integrating AZ information into information extraction tasks within materials science. They also propose testing a wider variety of models and embeddings.

Summary updated

BibTeX
@inproceedings{schrader2023mulms,
  author = {Timo Pierre Schrader and Teresa Bürkle and Sophie Henning and Sherry Tan and Matteo Finco and Stefan Grünewald and Maira Indrikova and Felix Hildebrand and Annemarie Friedrich},
  title = {{MuLMS-AZ: An Argumentative Zoning Dataset for the Materials Science Domain}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure