X-PuDu at SemEval-2022 Task 7: A Replaced Token Detection Task Pre-trained Model with Pattern-aware Ensembling for Identifying Plausible Clarifications

Junyuan Shang, Shuohuan Wang, Yu Sun, Yanjun Yu, Yue Zhou, Li Xiang, Guixiu Yang

TL;DR

The winning entry for SemEval-2022 Task 7 introduces X-PuDu, a system designed to identify plausible clarifications for underspecified phrases in instructional texts. By pairing a Replaced Token Detection (RTD) pre-trained backbone (DeBERTa-V3) with a retained pre-training projection head and a pattern-aware ensembling strategy, the system took 1st place in both multi-class classification and plausibility ranking by significant margins.

What Problem Does This Paper Address?

Instructional how-to guides (like articles on wikiHow) frequently rely on implicit, ambiguous, or underspecified steps. Human readers easily infer the missing details, but natural language processing (NLP) systems often struggle to distinguish between contextually plausible, neutral, or nonsensical clarifications.

SemEval-2022 Task 7 evaluated models on two objectives:

  1. SubTask-A (Multi-Class Classification): Classify a candidate clarification insertion into one of three classes: PLAUSIBLE, NEUTRAL, or IMPLAUSIBLE.
  2. SubTask-B (Ranking): Predict a continuous plausibility score on a 1.0 to 5.0 scale.

Standard pre-trained models trained with Masked Language Modeling (MLM) struggle here because MLM groups semantically related words together in vector space rather than discriminating whether an insertion actually makes contextual sense.

Methods

The authors' pipeline consists of four major components:

  1. Input Structuring: Each candidate clarification is inserted directly into the target sentence. The sample is formatted by concatenating contextual metadata: [Pattern, Title, Section Header, Previous Sentence, Target Sentence with filler, Follow-up Sentence].
  2. RTD-Based Backbone: Rather than MLM, the authors use DeBERTa-V3-Large, which is pre-trained via Replaced Token Detection (RTD). The RTD objective trains a discriminator to distinguish true original tokens from plausible generator-sampled replacements, mirroring the shared task's requirements.
  3. Reusing the Pre-Trained Head: Instead of discarding the pre-trained language modeling head during fine-tuning (the typical fine-tuning recipe), the authors keep it: $$\mathbf{H}_p = \text{LayerNorm}(\text{Act}(\mathbf{H}_b \mathbf{W}_1 + \mathbf{b}_1))$$ Token representations belonging to the candidate filler phrase are mean-pooled and passed through an enhanced multi-layer perceptron (Dropout $\to$ Dense $\to$ Tanh $\to$ Dropout).

  4. Task Heads:

  5. SubTask-A: Softmax over 3 classes, trained with cross-entropy.
  6. SubTask-B: A single regression unit squashed via Sigmoid and linearly scaled ($\text{Sigmoid}(\cdot) \times 4 + 1$) to force predictions strictly into the $[1.0, 5.0]$ range, trained with Mean Squared Error.
  7. Pattern-Aware Ensembling: Recognizing that different hyperparameter configurations excel on different linguistic patterns (due to label shifts across patterns), the authors ensemble model checkpoints conditionally based on the sample's linguistic pattern category.

Datasets

The authors exclusively used the official SemEval-2022 Task 7 dataset (derived from wikiHow revisions) without external data. The dataset is balanced across four structural linguistic patterns:

  • Added Compound
  • Fused Head
  • Implicit Reference
  • Metonymic Reference

Data splits:

  • Train: 19,975 samples (~5,000 per pattern)
  • Validation: 2,500 samples (625 per pattern)
  • Test: 2,500 samples (625 per pattern)

Key Insights and Ideas

  • RTD aligns with clarification verification: MLM pushes semantic neighbors closer together in the representation space, whereas RTD is explicitly penalized if it cannot detect subtly incorrect token swaps. This makes RTD representations naturally suited for detecting implausible phrase insertions.
  • The pre-training head contains useful signals: Retaining the projection layer directly above the transformer backbone preserved learned discriminative features, yielding a +1.29% accuracy boost over discarding it.
  • Patterns suffer from distinct label imbalances: Visualizing class distributions revealed that Fused Head contains very few NEUTRAL labels, while PLAUSIBLE strongly dominates Added Compound and Implicit Reference. Models tuned with different hyperparameters overfit or underfit these distributions differently; grouping ensemble weights by linguistic pattern yielded major performance gains.

Key Results

X-PuDu secured 1st place across both competition tracks:

Track Metric X-PuDu Standard Ensemble 2nd Place Human Upper Bound
SubTask-A (Classification) Accuracy 68.90% 66.50% 66.10% 77.10%
SubTask-B (Ranking) Spearman's $\rho$ 0.8070 0.7950 0.7850 0.8900

Ablation on the dev set showed DeBERTa-V3-Large scoring 67.25% accuracy, outperforming MLM-based alternatives like ERNIE (61.73%) and XLM-R Large (61.14%). Pattern-aware ensembling boosted dev accuracy from 71.08% to 75.20%.

Limitations and Future Work

  • Room for Improvement: A notable gap remains between the winning system's scores (68.90% accuracy, 0.8070 rank correlation) and estimated human upper bounds (77.1% accuracy, 0.89 rank correlation).
  • Model Scale: The study was limited to existing large-scale discriminator weights. The authors point to training RTD objectives on ultra-large foundation models (tens or hundreds of billions of parameters) as a promising avenue for closing the remaining gap with human performance.

Summary updated

Abstract

This paper describes our winning system on SemEval 2022 Task 7: Identifying Plausible Clarifications ofImplicit and Underspecified Phrases in Instructional Texts. A replaced token detection pre-trained model is utilized with minorly different task-specific heads for SubTask-A: Multi-class Classification and SubTask-B: Ranking. Incorporating a pattern-aware ensemble method, our system achieves a 68.90% accuracy score and 0.8070 spearman’s rank correlation score surpassing the 2nd place with a large margin by 2.7 and 2.2 percent points for SubTask-A and SubTask-B, respectively. Our approach is simple and easy to implement, and we conducted ablation studies and qualitative and quantitative analyses for the working strategies used in our system.
BibTeX
@inproceedings{shang2022x,
  author = {Junyuan Shang and Shuohuan Wang and Yu Sun and Yanjun Yu and Yue Zhou and Li Xiang and Guixiu Yang},
  title = {{X-PuDu at SemEval-2022 Task 7: A Replaced Token Detection Task Pre-trained Model with Pattern-aware Ensembling for Identifying Plausible Clarifications}},
  booktitle = {SemEval @ NAACL},
  year = {2022},
  url = {https://openreview.net/forum?id=46LdxEiiGZ},
}

Loading related content...

Figure