UnImplicit Shared Task Report: Detecting Clarification Requirements in Instructional Text

Michael Roth, Talita AnthonioACL 2021

TL;DR

This paper introduces a shared task focused on automatically identifying sentences in instructional texts that require clarification due to implicit or underspecified language. The best-performing models achieved 68.4% accuracy, demonstrating the feasibility of this task, though significant challenges remain, particularly with subtle meaning changes.

What Problem Does This Paper Address?

Instructional texts, like those found in wikiHow articles, often contain sentences that are technically correct but might be unclear to a reader due to implicit assumptions or underspecified details. For example, a sentence might say "Add the ingredients" without specifying which ingredients or how much. Ensuring clarity in such texts is crucial for effective instruction following. This paper addresses the challenge of automatically detecting these "clarification requirements" in sentences, aiming to build NLP systems that can flag potentially ambiguous instructions for revision. This is a step towards more robust and user-friendly instructional content generation and analysis.

Methods

The core of the methodology involves creating a dataset of sentences from instructional texts that either require clarification or do not. To identify sentences requiring clarification, the authors leveraged revision histories from wikiHowToImprove. They specifically filtered for edits that added clarifying information, distinguishing them from simple grammar corrections. To create a balanced dataset, each sentence flagged for clarification was paired with a sentence that remained unchanged over multiple revisions. The task for participating systems was a binary classification: predict whether a given sentence requires clarification. The authors also explored syntactic similarity as a way to pair clarification-requiring sentences with non-clarification sentences to mitigate potential biases.

Datasets

The primary dataset used is derived from wikiHowToImprove, a resource containing revision histories of wikiHow articles. This dataset was split into training, development, and test sets. Additionally, an out-of-domain dataset was created from board game manuals to test the generalizability of the models. This out-of-domain set was processed using the same criteria as the wikiHow data, with a slight relaxation on exact syntactic pattern matching to include more diverse clarification types.

Key Insights and Ideas

  • Transformer models are strong baselines: The top-performing systems heavily relied on transformer-based architectures, highlighting their effectiveness for this task.
  • Subtlety is the enemy: Edits that introduce subtle, context-specific shades of meaning (like adding modifiers) were significantly harder for models to detect compared to resolving pronoun references or 'do' verb replacements.
  • Syntactic similarity is a useful pairing strategy: Using syntactic similarity to pair clarification-requiring sentences with non-clarification sentences helped create a more balanced and representative dataset.
  • Out-of-domain performance drops significantly: Models trained on wikiHow data struggled when applied to board game manuals, suggesting that the nature of implicit language can vary across domains.
  • Commonsense knowledge is key: The difficulty in predicting certain types of clarifications (e.g., modifiers) suggests that human commonsense knowledge plays a crucial role that is hard for current models to capture.

Key Results

The shared task saw two participating teams. The best-performing system, Wiriyathammabhum, achieved an accuracy of 68.4% on the test set. This significantly outperformed a random baseline (50%) and a logistic regression baseline (62.3%). However, performance on the out-of-domain dataset dropped to 59.1%, falling below the logistic regression baseline. Per-type analysis revealed that systems were highly accurate (over 90%) in identifying pronoun replacements and 'do' verb replacements, but struggled with modifiers and quantifiers/modals, with accuracies around 50-55%.

Limitations and Future Work

The authors acknowledge that the distribution of revision types in the dataset was unbalanced, with modifier insertions being the most common type requiring clarification. They also note that the out-of-domain data showed a different distribution of revision types, primarily involving adverbial/adjectival modifiers or optional complements. Future work could explore more diverse revision types and consider richer contextual information, both in terms of linguistic context and the methods used. A follow-up task for SemEval-2022 is planned, focusing on predicting plausible clarification changes rather than just the need for clarification. The authors also suggest that understanding the "realistic upper bound" for this task and the conditions under which a revision is truly mandatory remain open questions.

Summary updated

BibTeX
@inproceedings{roth2021unimplicit,
  author = {Michael Roth and Talita Anthonio},
  title = {{UnImplicit Shared Task Report: Detecting Clarification Requirements in Instructional Text}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure