TA-MAMC at SemEval-2021 Task 4: Task-adaptive Pretraining and Multi-head Attention for Abstract Meaning Reading Comprehension

Jing Zhang, Yimeng Zhuang, Yinpei SuACL 2021

TL;DR

This paper tackles the challenge of abstract meaning reading comprehension by proposing a novel approach that combines task-adaptive pretraining with a multi-head attention classifier. Their method achieved top rankings in the SemEval-2021 Task 4, demonstrating the effectiveness of adapting pre-trained language models to specific downstream tasks.

What Problem Does This Paper Address?

Machine Reading Comprehension (MRC) is a crucial task for evaluating a machine's ability to understand human language. While extractive MRC (finding exact answer spans) has seen significant progress, abstract meaning comprehension, particularly in a multiple-choice format, remains challenging. This is partly due to the difficulty of understanding abstract concepts and the limited availability of large, labeled datasets for training deep neural networks. The SemEval-2021 Task 4 specifically focuses on this abstract meaning comprehension, requiring models to understand non-concrete and generalized concepts. The paper aims to improve performance on this task by developing a more effective pre-training and fine-tuning strategy.

Methods

The core of the proposed system lies in two main components:

  • Task-Adaptive Pretraining: Instead of relying solely on general-purpose pre-trained language models (LMs), the authors adapt the ELECTRA model using in-domain data. They leverage news abstractive summarization datasets (XSUM and NEWSROOM) as a source for this pretraining. To create a suitable dataset for abstract meaning comprehension, they regenerate the ReCAM task's multiple-choice format from these summarization datasets. This involves identifying abstract concepts and generating both gold and pseudo options.
  • Multi-head Attention Multiple-Choice Classifier (MAMC): A multi-head attention mechanism is employed on top of the pre-trained encoder. This classifier is designed to capture complex interactions between the query and context, ultimately predicting the correct answer from the given options. They also incorporate Wrong Answer Ensemble (WAE) for further performance enhancement.

Datasets

  • ReCAM: The primary dataset for evaluation, specifically designed for the SemEval-2021 Task 4, focusing on abstract meaning reading comprehension. It consists of news articles and multiple-choice questions.
  • XSUM: A dataset of BBC articles and professionally written single-sentence summaries, used for task-adaptive pretraining.
  • NEWSROOM: A large dataset of news articles and summaries, also utilized for task-adaptive pretraining.

Key Insights and Ideas

  • Pre-trained LM Discrepancy: The paper highlights that different pre-trained LMs (BERT, ALBERT, ELECTRA) exhibit significant performance variations on the abstract meaning MRC task, suggesting that the pre-training objective and dataset play a crucial role in downstream performance.
  • ELECTRA's Advantage: ELECTRA, with its discriminator-based pre-training objective (predicting replaced tokens), shows superior performance compared to BERT and ALBERT. This is attributed to its ability to learn more about unreasonable co-occurrences, which aids in understanding implicit semantic relations.
  • In-Domain Pretraining is Key: Task-adaptive pretraining on news summarization data significantly boosts performance on the ReCAM task, underscoring the importance of domain relevance in transfer learning.
  • Contrastive Learning Limitations: Interestingly, contrastive learning objectives (triplet and n-tuple loss) and NLI tasks, while popular in other NLP tasks, degrade performance on this specific abstract meaning MRC task, suggesting they are not suitable here.

Key Results

The proposed TA-MAMC system achieved top rankings in the SemEval-2021 Task 4:

  • Subtask 1: 1st place with a test accuracy of 95.11%.
  • Subtask 2: 2nd place with a test accuracy of 94.89%.

Task-adaptive pretraining alone improved dev accuracy by approximately 2.8-3.1% over direct fine-tuning. Adding the MAMC classifier further boosted performance, and WAE provided marginal gains.

Limitations and Future Work

  • Fail Case Analysis: The authors identified several categories of failure, including easy cases (ECO), complicated coreferences (EC1), complex reasoning (EC2), external knowledge dependency (EC3), and ambiguity (EC4). Ambiguity (EC4) and external knowledge dependency (EC3) were the most prevalent failure modes.
  • Data Augmentation Quality: While data augmentation was explored, it did not significantly improve performance, suggesting that automatically generated samples might be of lower quality than human-labeled data.
  • Future Work: The paper implicitly suggests further exploration into understanding and mitigating the identified failure cases, particularly those requiring external knowledge or complex reasoning. Investigating more sophisticated methods for generating high-quality pseudo-options for pretraining could also be a fruitful direction.

Summary updated

BibTeX
@inproceedings{zhang2021ta,
  author = {Jing Zhang and Yimeng Zhuang and Yinpei Su},
  title = {{TA-MAMC at SemEval-2021 Task 4: Task-adaptive Pretraining and Multi-head Attention for Abstract Meaning Reading Comprehension}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure