We present the results of the Machine Reading for Question Answering (MRQA) 2019 shared task on evaluating the generalization capabilities of reading comprehension systems. In this task, we adapted and unified 18 distinct question answering datasets into the same format. Among them, six datasets were made available for training, six datasets were made available for development, and the final six were hidden for final evaluation. Ten teams submitted systems, which explored various ideas including data sampling, multi-task learning, adversarial training, and ensembling. The best system achieved an average F1 score of 72.5 on the 12 held-out datasets, 10.7 absolute points higher than our initial baseline based on BERT.
Introduction
The 2nd Workshop on Machine Reading for Question Answering (MRQA), held at EMNLP 2019, addressed a fundamental limitation in current Machine Reading Comprehension (MRC) models: the lack of robustness to domain shift. While state-of-the-art systems perform exceptionally well when test data mirrors training data, their performance often degrades significantly when applied to unseen text distributions. The MRQA 2019 Shared Task was established to benchmark and improve the generalization capabilities of these systems across diverse domains.
The Challenge of Generalization
Generalization in MRC requires a model to extrapolate reasoning patterns learned from one source (e.g., Wikipedia) to entirely different contexts, such as biomedical abstracts, news articles, or literary texts. To evaluate this, the organizers unified 18 existing datasets into a consistent extractive format, where the goal is to identify the specific text span containing the answer to a given question.
Methodology and Participating Systems
Participating teams employed a variety of techniques to enhance model robustness:
Advanced Pre-trained Architectures: Systems moved beyond the standard BERT-Large baseline, leveraging newer models like XLNet and ERNIE 2.0.
Multi-task and Adversarial Training: Teams used auxiliary tasks and domain discriminators to encourage the learning of domain-invariant linguistic features.
Negative Sampling: Incorporating text segments that do not contain the answer helped models avoid over-reliance on simple keyword matching.
Ensembling: Combining predictions from multiple models was a common strategy to improve overall stability and accuracy.
Dataset Curation
The 18 datasets were categorized into three splits to rigorously test generalization:
Training Split: Six large datasets, including SQuAD, NewsQA, and Natural Questions.
Development Split: Six datasets for tuning, such as BioASQ (biomedical) and DROP (discrete reasoning).
Hidden Test Split: Six entirely hidden datasets for final evaluation, including BioProcess and MCTest.
Key Results and Insights
The shared task yielded several critical findings:
Top Performance: The D-Net system from Baidu secured first place with an average F1 score of 72.5 across the 12 held-out datasets, outperforming the BERT-Large baseline by 10.7 absolute points.
Baidu's D-Net system achieved the highest F1 score in the shared task.
Model Selection: The choice of pre-trained language model was the most significant factor in performance, with XLNet providing substantial gains in generalization.
Robustness Correlation: A strong positive correlation exists between in-domain and out-of-domain performance, suggesting that better base models are inherently more robust.
Remaining Challenges: Despite progress, a significant gap persists between machine performance and human-level reasoning, particularly in tasks requiring complex logic or arithmetic (e.g., DROP, ROPES).
Conclusion
The MRQA 2019 Shared Task highlights the progress made in building more generalizable MRC systems while underscoring the remaining challenges. Future work should focus on moving beyond extractive formats, improving evaluation metrics, and enhancing models' ability to handle long-context reasoning and implicit knowledge.
Summary updated
Abstract
We present the results of the Machine Reading for Question Answering (MRQA) 2019 shared task on evaluating the generalization capabilities of reading comprehension systems. 1 In this task, we adapted and unified 18 distinct question answering datasets into the same format.Among them, six datasets were made available for training, six datasets were made available for development, and the final six were hidden for final evaluation.Ten teams submitted systems, which explored various ideas including data sampling, multi-task learning, adversarial training, and ensembling.The best system achieved an average F1 score of 72.5 on the 12 held-out datasets, 10.7 absolute points higher than our initial baseline based on BERT.
BibTeX
@inproceedings{fisch2019proceedings,
author = {Adam Fisch and Alon Talmor and Robin Jia and Minjoon Seo and Eunsol Choi and Danqi Chen and Jonathan Berant and Vivek Srikumar and Pei‐Chun Chen and Abby Vander Linden and Brittany Harding and Aniruddha Kembhavi and Dustin Schwenk and Jonghyun Choi and Ali Farhadi and Tom Kwiatkowski and Jennimaria Palomaki and Michael Collins and Ankur P. Parikh and Chris Alberti and Danielle Epstein and Illia Polosukhin and Matthew Kelcey and Jacob Devlin and Kenton Lee and Kristina Toutanova and Llion Jones and Ming‐Wei Chang and Andrew M. Dai and Jakob Uszkoreit and Christopher D Manning and Hangbo Bao and Dong Li and Furu Wei and Wenhui Wang and Nan Yang and Lei Cui and Songhao Piao and Ming Zhou and Rupesh K. Srivastava and Klaus Greff and Jürgen Schmidhuber and Ashish Vaswani and Noam Shazeer and Niki Parmar and Frédéric Béchet and Cindy Aloui and Delphine Charlet and Géraldine Damnati and Johannes Heinecke and Alexis Nasr and Frédéric Herledan and Frdric Herldan and Petr Baudi and Jan ediv and Adina Williams and Nikita Nangia and Samuel Bowman and Zhilin Yang and Zihang Dai and Yiming Yang and Jaime Carbonell and Ruslan Salakhutdinov and Quoc Le and Zhilin Yang and Peng Qi and Saizheng Zhang and Yoshua Bengio and William Cohen and Ruslan Salakhutdinov and Christopher Manning and Zhengyan Zhang and Xu Han and Zhiyuan Liu and Xin Jiang and Maosong Sun and Qun Liu and Steven Bird and Edward Loper and Danqi Chen and Adam Fisch and Jason Weston and Antoine Bordes and Peter Clark and Oren Etzioni and Tushar Khot and Ashish Sabharwal and Oyvind Tafjord and Peter Turney and Daniel Khashabi and Jacob Devlin and Ming-Wei Chang and Kenton Lee and Kristina Toutanova and Minghao Hu and Yuxing Peng and Zhen Huang and Nan Yang and Ming Zhou and Panayiota Kendeou},
title = {{Proceedings of the 2nd Workshop on Machine Reading for Question Answering}},
booktitle = {Openalex Unknown},
year = {2019},
doi = {10.18653/v1/d19-58},
}