DuluthNLP at SemEval-2022 Task 7: Classifying Plausible Alternatives with Pre–trained ELECTRA

Samuel Akrah, Ted PedersenNAACL 2022

TL;DR

This paper tackles the challenge of automatically identifying plausible replacements for missing words in instructional texts, a crucial step for improving clarity and usability. The DuluthNLP team achieved a respectable 6th place in the SemEval-2022 Task 7 by fine-tuning the ELECTRA language model, demonstrating the effectiveness of pre-trained models with careful hyperparameter tuning, particularly class weights, for this nuanced NLP task.

What Problem Does This Paper Address?

Instructional texts, like those found on WikiHow, are vital for guiding users through tasks. However, these texts can sometimes contain underspecified or implicitly missing phrases, leading to confusion and requiring revisions. SemEval-2022 Task 7 specifically addresses this by asking models to classify or rank potential "fillers" for these omitted tokens based on their plausibility within the given context. This is a challenging problem because determining plausibility requires a deep understanding of sentence structure, context, and the subtle nuances of language, going beyond simple word co-occurrence. The task is divided into two subtasks: Subtask A, which classifies fillers into IMPLAUSIBLE, NEUTRAL, or PLAUSIBLE categories, and Subtask B, which ranks fillers on a 1-5 scale. The DuluthNLP team focused on Subtask A.

Methods

The DuluthNLP system explored several pre-trained language models, including BERT, RoBERTa, and ELECTRA, for Subtask A. These models were fine-tuned on the provided training data. The core methodology involved treating the problem as a classification task where the model predicts the plausibility category for each potential filler.

A key aspect of their approach was the use of ELECTRA (Efficiently Learning an Encoder that Classifies Token Replacements Accurately). Unlike BERT's Masked Language Modeling (MLM) which masks tokens and predicts the original, ELECTRA uses a "Replaced Token Detection" objective. In this setup, a generator model creates plausible but synthetic replacements for masked tokens, and a discriminator model then learns to distinguish between the original tokens and these generated fakes. The authors highlight that ELECTRA's approach is more computationally efficient and can match or exceed BERT's performance.

Furthermore, the team experimented with class weights to address an initial skew in their model's predictions towards the "NEUTRAL" category on the development set. By assigning specific weights to penalize NEUTRAL predictions, they aimed to improve the model's accuracy and better reflect the actual distribution of labels. They also explored highlighting the filler with a special "[filler]" token, though this did not yield consistent gains on the official test data.

Datasets

The primary dataset used was provided by the organizers of SemEval-2022 Task 7. This dataset is derived from the WikiHowToImprove Corpus, which consists of edits from a large number of WikiHow sentences. Each instance in the dataset includes an article title, section header, and a sentence with a blank token, along with five potential fillers. These fillers are annotated with plausibility labels (IMPLAUSIBLE, NEUTRAL, PLAUSIBLE) for Subtask A and a 1-5 score for Subtask B. The authors also mention using development data for hyperparameter tuning and evaluation.

Key Insights and Ideas

  • ELECTRA's Advantage: The paper strongly suggests that ELECTRA's "Replaced Token Detection" pre-training objective is particularly well-suited for tasks like plausibility classification, outperforming BERT and RoBERTa in some configurations. Its ability to learn from all tokens in a sequence, rather than just masked ones, seems to be a significant factor.
  • The Power of Class Weights: Addressing class imbalance, even when the training data appears balanced, is crucial. The initial model's tendency to predict "NEUTRAL" was a significant issue. Applying carefully tuned class weights was instrumental in improving accuracy and correcting this bias.
  • Subtlety of "IMPLAUSIBLE": The error analysis revealed a persistent difficulty in correctly classifying "IMPLAUSIBLE" labels. This suggests that identifying what is definitively wrong or nonsensical is more challenging for the model than identifying what is merely neutral or plausible.

Key Results

  • The DuluthNLP system, using the ELECTRA model with tuned class weights, achieved an accuracy of 53.3% on the official evaluation test data for Subtask A.
  • This result placed them 6th out of 8 systems in the task.
  • On the development data, ELECTRA with class weights achieved an accuracy of 55.6%, while RoBERTa with class weights achieved 55.2%, and BERT with class weights achieved 52.2%.
  • Without class weight tuning, the models performed significantly worse, with ELECTRA achieving only 44.3% accuracy on the development data.
  • A baseline Logistic Regression model achieved 34.8% accuracy on the development data.

Limitations and Future Work

  • Accuracy Ceiling: The authors acknowledge that the achieved accuracy of "at best, a little above 50%" is not high enough for deployment in real-world scenarios without human oversight, as inaccurate clarifications could lead to harm.
  • "IMPLAUSIBLE" Classification Difficulty: The model struggled to accurately identify IMPLAUSIBLE labels, indicating a need for further research into this specific aspect of the task.
  • Out-of-Distribution Generalization: The model's performance on data outside of the training distribution is not guaranteed, as it relies on contextual representations learned from the specific dataset.
  • Ethical Considerations: The paper touches upon the significant computational and environmental costs of training large language models. They mitigated this by fine-tuning base models rather than larger variants. They also discuss the potential for models to perpetuate biases present in their training data and the need for human verification of model outputs.
  • Future Work Suggestions: The authors mention the potential of "Reflection Machines" – decision support systems that compel users to justify their choices – as a way to mitigate risks associated with automated suggestions. They also suggest that further research into improving the classification of IMPLAUSIBLE labels is warranted.

Summary updated

BibTeX
@inproceedings{akrah2022duluthnlp,
  author = {Samuel Akrah and Ted Pedersen},
  title = {{DuluthNLP at SemEval-2022 Task 7: Classifying Plausible Alternatives with Pre–trained ELECTRA}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
}
Figure