TL;DR
This paper introduces SemEval-2022 Task 7, a new challenge focused on automatically identifying plausible clarifications for underspecified phrases in instructional texts. The task aims to improve how well machines understand and generate helpful explanations, crucial for making instructions clearer and more robust.
What Problem Does This Paper Address?
Natural language, especially in instructional texts, often relies on implicit references and underspecified phrases. This ambiguity can lead to misunderstandings and errors when following instructions. While humans naturally infer missing information, machines struggle with this. The paper addresses the challenge of automatically identifying and evaluating the plausibility of clarifications that could resolve these ambiguities, moving beyond simple cloze-style prediction to assess the quality and appropriateness of potential explanations.
Methods
The core of this work is a novel shared task designed to evaluate NLP systems' ability to distinguish plausible clarifications from implausible ones. The methodology draws inspiration from cloze-style tasks but extends them to assess clarification plausibility.
- Data Creation: The dataset was built upon revision histories from wikiHow instructional texts. This involved:
- Pattern-based extraction: Identifying revisions that inserted phrases to clarify implicit references or underspecified parts of sentences.
- Generative clarification generation: Using transformer-based language models (BERT) to generate alternative clarifications for these identified instances.
- Human plausibility annotation: Collecting human judgments on the plausibility of both the original (human-inserted) clarification and the generated alternatives.
- Task Design: Two subtasks were defined:
- Classification: Systems predict one of three labels: IMPLAUSIBLE, NEUTRAL, or PLAUSIBLE.
- Ranking: Systems predict a continuous plausibility score.
Datasets
The primary dataset used is wikiHowToImprove, a resource containing revision histories from over 250,000 how-to guides. This dataset was used to extract instances of clarifications. A new benchmark dataset was created specifically for this shared task, comprising manually annotated clarifications and their plausibility judgments. This dataset includes approximately 25,000 instances for training, development, and testing, covering four specific phenomena: implicit references, fused heads, noun compounds, and metonymy.
Key Insights and Ideas
- The value of revision histories: Instructional text revision histories are a rich source for identifying and studying how language is clarified.
- Beyond single correct answers: Unlike traditional cloze tasks, this work acknowledges that multiple clarifications can be plausible, reflecting real-world ambiguity and potential misunderstandings.
- Generative models can produce good clarifications: The study found that automatically generated clarifications can be as good as, or even better than, human-inserted ones in some cases.
- The "NEUTRAL" label is challenging: Human annotators found it difficult to consistently distinguish between NEUTRAL and other labels, and systems also struggled with this category. This highlights the nuanced nature of plausibility.
- Identifying multiple plausible clarifications is a distinct challenge: The ability to detect when a context allows for multiple plausible clarifications is a valuable capability, potentially revealing areas where instructions are inherently ambiguous.
Key Results
- Task 1 (Classification): The winning team, X-PuDu, achieved an accuracy of 68.9%, which is about 10.5 percentage points below the human upper bound of 79.4%. This indicates the task is challenging but solvable by current NLP models.
- Task 2 (Ranking): The top system achieved a Spearman's rank correlation coefficient of 0.807, demonstrating good performance in ranking clarifications by plausibility.
- Additional Evaluation (Multiple Plausible Clarifications): The best system could identify contexts with multiple plausible clarifications with an accuracy of 75.2%.
- Impact of NEUTRAL label: Systems performed better when the NEUTRAL label was excluded from evaluation, suggesting that the ambiguity captured by this label is particularly hard for models to handle.
Limitations and Future Work
- Limited phenomena: The task currently focuses on only four specific types of clarifying phenomena. Future work could expand to include more forms of clarification.
- Individual assessment: Clarifications were assessed individually, without considering how they might interact or conflict with each other. Future research should investigate scenarios where multiple incompatible clarifications exist and how this reveals sources of misunderstanding.
- Dataset size: The training set, with less than 20,000 instances, is relatively small.
- NEUTRAL label ambiguity: The significant proportion of NEUTRAL labels in the dataset and the difficulty in distinguishing them present an ongoing challenge.
The authors suggest that combining complementary contributions from different participating teams could lead to further improvements. Future work should also explore individual differences in how people interpret language and the circumstances under which multiple clarifications are genuinely incompatible.
Summary updated