TTCB System Description to a Shared Task on Implicit and Underspecified Language 2021

Peratham WiriyathammabhumACL 2021

TL;DR

This paper presents a Transformer-based system (TTCB) for predicting revision requirements in collaboratively edited instructions. The authors explore various Transformer models, training strategies, and loss functions, finding that XLNet with a linear annealing scheduler and cross-entropy loss achieves the best performance, highlighting the challenges in understanding implicit language.

What Problem Does This Paper Address?

The paper tackles the problem of identifying sentences in instructional texts that require revisions due to implicit or underspecified language. This is crucial for improving the clarity and usability of collaboratively edited content, such as wikiHow instructions. The task involves classifying sentences based on whether they need revision, with specific revision types like pronoun replacement or adverbial modification. Existing work has explored models like BERT and BiLSTM, but this paper aims to further investigate state-of-the-art Transformer models for this nuanced task.

Methods

The core of the TTCB system relies on Transformer-based language models. The authors experimented with several models, including XLNet, OpenGPT-2, and BigBird, all of which are considered state-of-the-art for text classification. They explored different training schemes, including linear annealing schedulers, and loss functions like cross-entropy and cost-sensitive cross-entropy. Data augmentation techniques were also investigated, specifically by labeling revised sentences as "no revision required" to increase the number of negative instances. Additionally, they explored Siamese training using sentence-Transformers for learning similarity embeddings.

Datasets

The primary dataset used is the wikiHowToImprove dataset. This dataset consists of sentences and their corresponding revisions, categorized into five rule-based revision types. The authors mention that their shared task subset contains 45,909 sentences in total, with 39,187 in the training set.

Key Insights and Ideas

  • XLNet as a Strong Baseline: XLNet, a generalized autoregressive pretraining model, proved to be a robust choice for this task, outperforming other Transformer models in their experiments.
  • Limited Gains from Design Choices: Surprisingly, many explored design choices, such as different loss functions or data augmentation, did not yield significant improvements in validation metrics. This suggests the task itself presents a substantial challenge for current models.
  • MiniLM's Efficiency: The MiniLM model offered a comparable F1 score to XLNet but at significantly reduced training costs, making it a more efficient option for certain applications.
  • Siamese Training's Recall Advantage: Siamese-BERT models showed high F1 scores, potentially due to their inherent recall advantage stemming from the nature of embedding vector spaces.
  • Attention Visualization Insights: Visualizing XLNet's attention heads revealed that early layers capture local patterns, middle layers learn longer dependencies, and top layers seem to focus on revision-specific cues, though this was not always definitive.

Key Results

The best performing model, an XLNet model with a linear annealing scheduler and cross-entropy loss, achieved a test accuracy of 68.45% and a validation accuracy of 68.84%. While other models like BigBird and OpenGPT-2 were explored, XLNet consistently performed well. The MiniLM model achieved a validation accuracy of 68.44% and an F1 score of 71.72%, with a training time that was half that of XLNet. Data augmentation with negative class augmentation actually decreased accuracy to 64.74%, but adding cost-sensitive binary cross-entropy helped recover performance.

Limitations and Future Work

The authors acknowledge that the achieved accuracies (around 70%) are "not very practical," indicating a significant challenge in understanding implicit and underspecified language. They suggest several future directions:

  • Advanced Loss Functions: Exploring more sophisticated loss functions like Triplet loss for Siamese training.
  • Syntactic Information Integration: Investigating the integration of syntactic preprocessing and sentence embedding training, as suggested by recent work.
  • More Data Analysis: Conducting further data analysis to pinpoint specific learning requirements for models on this task.
  • Exploring Other Models: While they focused on XLNet, OpenGPT-2, and BigBird, other advanced Transformer architectures could be explored.

Summary updated

BibTeX
@inproceedings{wiriyathammabhum2021ttcb,
  author = {Peratham Wiriyathammabhum},
  title = {{TTCB System Description to a Shared Task on Implicit and Underspecified Language 2021}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure