X-PuDu at SemEval-2022 Task 7: A Replaced Token Detection Task Pre-trained Model with Pattern-aware Ensembling for Identifying Plausible Clarifications

Junyuan Shang, Shuohuan Wang, Yu Sun, Yanjun Yu, Yue Zhou, Li Xiang, Guixiu Yang

Abstract

Instructional texts often contain implicit or underspecified phrases that require clarification for full comprehension. This paper presents the X-PuDu system, the winning solution for SemEval-2022 Task 7: "Identifying Plausible Clarifications of Implicit and Underspecified Phrases in Instructional Texts." Our approach leverages Replaced Token Detection (RTD) pre-trained models, specifically DeBERTa-V3, combined with task-specific heads for multi-class classification and ranking. By incorporating a novel pattern-aware ensemble method, our system achieved a 68.90% accuracy score in SubTask-A and a 0.8070 Spearman's rank correlation score in SubTask-B, outperforming the second-place team by significant margins. We provide a detailed analysis of our methodology, including the benefits of reusing pre-trained language modeling heads and the efficacy of pattern-specific ensembling.

Introduction

In instructional manuals and "how-to" guides, authors frequently omit information that is considered common knowledge or contextually implied. While humans navigate these gaps effortlessly, machine understanding of such texts remains a challenge. SemEval-2022 Task 7 focuses on this problem by requiring models to evaluate the plausibility of clarifications for ambiguous phrases in wikiHow articles.

The task is divided into two components:

  1. Subtask A (Multi-class Classification): Categorizing a clarification as Plausible, Neutral, or Implausible.
  2. Subtask B (Ranking): Assigning a continuous plausibility score (1.0 to 5.0) to rank potential clarifications.

The X-PuDu system addresses these challenges by moving beyond standard generative pre-training objectives toward discriminative models that are better suited for fine-grained semantic judgment.

The Challenge of Instructional Ambiguity

Standard Natural Language Processing (NLP) models are typically trained using Masked Language Modeling (MLM), where the model predicts a hidden token. While effective for general representation, MLM tends to cluster semantically similar words together in vector space. In the context of clarification, however, the model must distinguish between words that are semantically related but contextually inappropriate. For example, in an instruction to "clean the surface," a clarification suggesting "table" might be plausible, while "floor" might be implausible given the specific context, even though both are surfaces. This requires a high degree of discriminative precision.

Methodology: Replaced Token Detection

Our core strategy involves the use of Replaced Token Detection (RTD) models, specifically DeBERTa-V3. Unlike MLM, which is a generative task, RTD is a discriminative pre-training objective where the model learns to identify whether a token is the original word or a replacement generated by a small auxiliary model.

This "discriminator" mindset is inherently aligned with the SemEval task: the model is already trained to spot "impostor" tokens that do not fit the context. By fine-tuning this discriminator for plausibility classification, we leverage the model's existing ability to detect subtle contextual inconsistencies.

Reusing the Pre-trained Head

A common practice in fine-tuning is to discard the pre-trained output layer (the "head") and replace it with a fresh task-specific layer. However, we found that the pre-trained RTD head contains critical information for distinguishing similar tokens. By preserving and reusing this head during fine-tuning, we maintained the nuanced semantic boundaries established during large-scale pre-training, leading to a measurable boost in performance.

Pattern-aware Ensembling

The SemEval-2022 dataset identifies four distinct linguistic patterns of ambiguity:

  • Added Compound: Clarifying a noun by adding a modifier.
  • Fused Head: Resolving a noun phrase where the head is missing.
  • Implicit Reference: Identifying an unstated object or entity.
  • Metonymic Reference: Resolving a figure of speech where a thing is called by the name of something associated with it.

Our analysis revealed that different model configurations and hyperparameters yielded varying levels of success across these patterns. To capitalize on this, we implemented a pattern-aware ensemble. Instead of a uniform average of all model outputs, we aggregated predictions separately for each linguistic category, allowing the system to rely on the most effective model for each specific type of ambiguity.

Experimental Results

The X-PuDu system secured first place in both subtasks of the competition.

  • Subtask A: Our system achieved 68.90% accuracy, a 2.7 percentage point lead over the runner-up.
  • Subtask B: We achieved a Spearman’s rank correlation of 0.8070, surpassing the second-place team by 2.2 percentage points.

An ablation study confirmed the importance of our architectural choices. Specifically, removing the pre-trained head caused accuracy to drop from 67.25% to 65.96% on the validation set, validating our hypothesis that the pre-trained discriminator's knowledge is vital for this task.

Conclusion

The success of the X-PuDu system demonstrates that discriminative pre-training objectives like RTD are exceptionally well-suited for tasks involving plausibility and clarification. While our system set a new state-of-the-art, a gap remains between machine performance (68.9%) and the estimated human upper bound (77.1%). Future work will focus on scaling these discriminative approaches and exploring how to better integrate complex instructional logic into the pre-training phase to further close this gap.

Summary updated

Abstract

This paper describes our winning system on SemEval 2022 Task 7: Identifying Plausible Clarifications of Implicit and Underspecified Phrases in Instructional Texts. A replaced token detection pre-trained model is utilized with minorly different task-specific heads for SubTask-A: Multi-class Classification and SubTask-B: Ranking. Incorporating a pattern-aware ensemble method, our system achieves a 68.90% accuracy score and 0.8070 spearman's rank correlation score surpassing the 2nd place with a large margin by 2.7 and 2.2 percent points for SubTask-A and SubTask-B, respectively. Our approach is simple and easy to implement, and we conducted ablation studies and qualitative and quantitative analyses for the working strategies used in our system.
BibTeX
@inproceedings{shang2022x,
  author = {Junyuan Shang and Shuohuan Wang and Yu Sun and Yanjun Yu and Yue Zhou and Li Xiang and Guixiu Yang},
  title = {{X-PuDu at SemEval-2022 Task 7: A Replaced Token Detection Task Pre-trained Model with Pattern-aware Ensembling for Identifying Plausible Clarifications}},
  booktitle = {SemEval @ NAACL},
  year = {2022},
  url = {https://openreview.net/forum?id=46LdxEiiGZ},
}
Figure