Ensemble ALBERT and RoBERTa for Span Prediction in Question Answering

Sony Bachina, Spandana Balumuri, Sowmya Kamath SACL 2021

TL;DR

This paper tackles the challenge of extractive question answering (QA) in document-grounded dialogues, specifically focusing on identifying the precise text span that answers a user's question. The authors leverage and ensemble powerful pre-trained transformer models like RoBERTa, ALBERT, and ELECTRA, achieving a strong performance on the DialDoc21 shared task. Their work highlights the effectiveness of combining diverse transformer architectures for improved accuracy in complex QA scenarios.

What Problem Does This Paper Address?

Question answering (QA) systems aim to retrieve relevant answers from large amounts of text. A particularly challenging sub-problem is extractive QA, where the answer is a contiguous span of text within a given document. This becomes even more complex in conversational settings, like dialogues, where the context is dynamic and the questions might refer to previous turns. The DialDoc21 shared task specifically focuses on "Knowledge Identification," which means pinpointing the exact text span within a document that answers the next agent's turn in a dialogue. This is crucial for building more natural and helpful chatbots and virtual assistants that can accurately extract information from provided context. The paper addresses the difficulty of accurately identifying these spans, especially in long documents and complex dialogue histories.

Methods

The core of the paper's approach involves fine-tuning pre-trained transformer models for the span prediction task. They experimented with three main families of models:

  • RoBERTa: A robustly optimized BERT variant that uses dynamic masking and other improvements for better pre-training.
  • ALBERT: Designed to reduce the number of parameters in BERT while maintaining performance, using techniques like embedding factorization and parameter sharing across layers.
  • ELECTRA: Employs a novel "replaced token detection" (RTD) pre-training task, which is more computationally efficient and often leads to better downstream performance.

To further boost performance, the researchers explored ensemble techniques. They combined the predictions of multiple fine-tuned models, using the confidence score from each model to determine the final prediction. This approach aims to leverage the strengths of different architectures and reduce variance. They also experimented with pre-training some of these models on additional datasets (SQuAD2.0, Natural Questions, CORD-19) before fine-tuning them on the DialDoc21 dataset.

Datasets

The primary dataset used for training and evaluation was the Doc2Dial dataset. This dataset consists of dialogues between an agent and an end-user, along with grounding documents from various social websites. The test dataset included an unseen COVID-19 related domain (cdccov19), which allowed for evaluation on out-of-domain performance.

Additionally, the models were pre-trained or fine-tuned on:

  • SQuADv2.0: A widely used dataset for extractive QA, including unanswerable questions.
  • Natural Questions (NQ): A dataset of real user questions posed to Google search.
  • CORD-19: A large corpus of COVID-19 research papers.

Key Insights and Ideas

  • Ensembling is Key: Combining predictions from different transformer models (RoBERTa, ALBERT, ELECTRA) significantly improved performance over individual models, demonstrating the benefit of diverse architectures.
  • Pre-training Matters: Pre-training models on a variety of relevant datasets (SQuAD2.0, NQ, CORD-19) before fine-tuning on the target DialDoc21 dataset led to better results, suggesting that broader exposure to QA-related data is beneficial.
  • ALBERT's Efficiency: ALBERT models, despite their parameter efficiency, showed competitive performance, especially when fine-tuned on larger datasets.
  • Out-of-Domain Robustness: The inclusion of an unseen COVID-19 domain in the test set allowed for assessing how well the models generalize to new topics, highlighting the importance of robust generalization.

Key Results

The paper reports strong performance on the DialDoc21 Knowledge Identification subtask. Their best ensemble model, Ensemble2 (combining ALBERT ensemble with RoBERTa base models fine-tuned on NQ and COVID datasets), achieved an Exact Match score of 58.58 and an F1 score of 73.39 on the test dataset. This performance placed their team (SB_NITK) 6th on the leaderboard. Individual models also showed promising results, with roberta-base-squadv2-nq achieving the highest Exact Match and F1 scores among the single models on the validation set.

Limitations and Future Work

The authors acknowledge that their approach focused on extractive QA. While they achieved good results, they plan to extend their approach and experiment with other ensembling techniques for further performance enhancement. They also aim to explore avenues for improved scalability when applied to larger datasets, which is a common challenge with large transformer models. The paper also implicitly suggests that further research into handling very long documents and more complex dialogue histories could yield even better results.

Summary updated

BibTeX
@inproceedings{bachina2021ensemble,
  author = {Sony Bachina and Spandana Balumuri and Sowmya Kamath S},
  title = {{Ensemble ALBERT and RoBERTa for Span Prediction in Question Answering}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure