niksss at SemEval-2022 Task7:Transformers for Grading the Clarifications on Instructional Texts

Nikhil SinghNAACL 2022

TL;DR

This paper presents a transformer-based system for automatically grading the plausibility of clarifications in instructional texts, achieving 7th place in SemEval-2022 Task 7. The authors explored both T5 and BERT models, ultimately finding BERT to be more efficient and effective for this text-classification and regression task.

What Problem Does This Paper Address?

Instructional texts, like recipes or assembly guides, can sometimes be ambiguous. To improve clarity, "clarifications" are added. The challenge is to automatically assess how good these clarifications are. SemEval-2022 Task 7 aimed to develop computational models that can predict the plausibility of these clarifications. This is crucial for improving the quality and usability of instructional content, especially in domains where clear instructions are paramount. The task was split into two subtasks: one classifying clarifications as plausible, implausible, or neutral, and another ranking them on a finer scale.

Methods

The authors experimented with two main transformer-based architectures: T5 and BERT.

  • T5 Model: Initially, they treated the problem as a text-to-text generation task using T5. The input to the T5 model was constructed by concatenating the article title, section header, previous context, the sentence with the placeholder for the clarification, and the follow-up context.
  • BERT Model: For Subtask A (classification), they switched to a BERT-based model. Here, the input was a continuous text sequence combining the article title, section header, previous context, sentence, and follow-up context. This sequence was then embedded into a 768-dimensional feature vector. This vector was passed through a dropout layer for regularization.
  • Subtask B (Regression): For the finer-grained ranking task, they adapted the BERT model used for Subtask A into a regression model.

The data was pre-processed and split into training (80%), validation (10%), and testing (10%) sets.

Datasets

The primary dataset used for this task is the wikiHowToImprove dataset (Anthonio et al., 2020). This dataset was adapted for SemEval-2022 Task 7, with variations from previous work. The task framed it as a cloze-style task where systems had to score how well potential "fillers" (clarifications) fit a given context.

Key Insights and Ideas

  • BERT's Efficiency: The BERT-based model proved to be significantly more efficient in terms of training and inference time compared to the T5 model, making it the preferred choice for their submission.
  • Classification vs. Regression: The task naturally split into classification (plausible/implausible/neutral) and regression (ranking), requiring different model adaptations.
  • Contextual Information is Key: The success of transformer models highlights the importance of capturing the surrounding context (article title, previous/follow-up sentences) to accurately judge the plausibility of a clarification.

Key Results

  • Subtask A (Classification): The BERT-based model achieved an accuracy of 44.40% on the hold-out test set, ranking 7th overall in the shared task. This was noted as being substantially above a naive majority class baseline and comparable to the task organizers' baseline.
  • Subtask B (Regression): The adapted BERT model achieved a Spearman's rank correlation coefficient (SRCC) of 0.25200, ranking 5th in this subtask.

Limitations and Future Work

  • Distinguishing Neutral/Plausible: The authors noted that their model struggled to distinguish between "neutral" and "plausible" clarifications, as the semantic difference can be subtle. This is exacerbated when the filler word is changed, leading to only minor changes in the feature vector.
  • Overfitting: There's a possibility that the model overfitted to the development set, which could have contributed to a decrease in performance on the test set.
  • Limited Context Window: The current transformer models used have a limited attention span, only considering a limited context around the filler. The authors suggest that models with longer context windows, such as BigBird or Longformer, could be more beneficial for this task.

Summary updated

BibTeX
@inproceedings{singh2022niksss,
  author = {Nikhil Singh},
  title = {{niksss at SemEval-2022 Task7:Transformers for Grading the Clarifications on Instructional Texts}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
}
Figure