PALI at SemEval-2022 Task 7: Identifying Plausible Clarifications of Implicit and Underspecified Phrases in Instructional Texts

Zhou Mengyuan, Dou Hu, Mengfei Yuan, Jin Zhi, Xiyang Du, Lianxin Jiang, Yang Mo, Xiaofeng ShiNAACL 2022

TL;DR

This paper introduces PALI, a system designed to identify plausible clarifications for implicit or underspecified phrases in instructional texts, a task that goes beyond traditional cloze tests by evaluating the absolute quality of potential fillers. Their approach combines state-of-the-art transformer models with novel techniques like grouped layer-wise learning rate decay and contrastive learning, achieving strong performance on both classification and regression subtasks.

What Problem Does This Paper Address?

Traditional cloze tests, common in NLP for evaluating discourse understanding, typically focus on predicting the single most likely word to fill a blank. However, SemEval-2022 Task 7 presents a more nuanced challenge: identifying and evaluating the plausibility of multiple potential clarifications for underspecified phrases within instructional texts. This task requires not just selecting the best filler but also assessing its inherent quality and how well it clarifies the original ambiguous statement. The paper addresses the need for NLP systems that can perform this deeper level of semantic understanding and quality assessment in the context of instructional content.

Methods

The PALI system is built upon an ensemble of pre-trained transformer models, specifically RoBERTa, XLM-RoBERTa, and DeBERTa. To enhance their performance on this task, the authors incorporate several key techniques:

  • Grouped Layer-wise Learning Rate Decay (GLLRD): This strategy applies different learning rates to different layers of the pre-trained models during fine-tuning. It assigns higher learning rates to top layers (which capture more specific information) and lower rates to bottom layers (which capture more general information), aiming to prevent catastrophic forgetting and improve fine-tuning.
  • Contrastive Learning Loss (NTXent Loss): Inspired by SimCLR, this loss function is added to maximize agreement between differently augmented views of the same data example in the latent space. This helps the model learn more robust sentence representations, crucial for evaluating the quality of various fillers.
  • Diverse Pooling Heads: Instead of relying solely on the standard CLS token for sentence representation, the authors experiment with different pooling strategies:
  • CLS Pooling Head: The standard approach using the CLS token.
  • Mean-Max Pooling Head: Concatenates mean and max pooling of the last hidden states.
  • LSTM plus Attention Pooling Head: Utilizes an LSTM followed by an attention module to process sequential information from hidden states.
  • Data Preprocessing Strategies: Two methods were explored for constructing input data. Strategy Two, which replaces resolved pattern labels with their textual explanations, proved more effective.

Datasets

The paper evaluates its system on the SemEval-2022 Task 7 dataset. This dataset is described as a complex cloze task involving instructional texts. It includes instances with article titles, section headers, previous and follow-up context, a sentence with a placeholder, five candidate fillers, and a gold score indicating the plausibility of each filler. The dataset is used for both training and evaluation.

Key Insights and Ideas

  • Beyond Simple Prediction: The task highlights that evaluating the absolute quality of potential clarifications is a distinct and challenging NLP problem, moving beyond just finding the most probable word.
  • Transformer Model Performance Discrepancy: The study reveals significant performance differences between various pre-trained transformer models (DeBERTa-v3 outperforming RoBERTa and XLM-RoBERTa).
  • Importance of Fine-tuning Strategies: Techniques like GLLRD and contrastive learning are shown to be crucial for adapting large pre-trained models to this specific, potentially few-shot, task, leading to substantial performance gains.
  • Pooling Head Matters: The choice of pooling head significantly impacts performance, with the LSTM plus Attention pooling head demonstrating superior results for capturing sentence-level semantics.
  • Data Representation is Key: Enriching the input by replacing categorical "resolved pattern" labels with their descriptive explanations improved performance, suggesting that richer contextual information aids the model.

Key Results

The PALI system achieved competitive results on the SemEval-2022 Task 7:

  • Subtask 1 (Classification): The system achieved a test accuracy of 0.654, ranking 4th on the leaderboard.
  • Subtask 2 (Regression): The system achieved a test Spearman's rank correlation coefficient of 0.785, ranking 2nd on the leaderboard.

Specifically, the DeBERTa-v3 model, combined with Strategy Two data construction, GLLRD, NTXent loss, and the LSTM plus Attention pooling head, yielded the best performance.

Limitations and Future Work

  • Limitations: The paper notes that the dataset is "few-sample," implying that performance might be sensitive to the limited training data. The authors also mention that while their system advances normal cloze task models, there's still room for improvement.
  • Future Work: The paper implicitly suggests exploring further model fusion techniques and potentially more advanced data augmentation or preprocessing strategies. The success of different pooling heads also opens avenues for investigating more sophisticated methods for sentence representation.

Summary updated

BibTeX
@inproceedings{mengyuan2022pali,
  author = {Zhou Mengyuan and Dou Hu and Mengfei Yuan and Jin Zhi and Xiyang Du and Lianxin Jiang and Yang Mo and Xiaofeng Shi},
  title = {{PALI at SemEval-2022 Task 7: Identifying Plausible Clarifications of Implicit and Underspecified Phrases in Instructional Texts}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
}
Figure