Proceedings of the 3rd Workshop on Machine Reading for Question Answering

Adam Fisch, Alon Talmor, Patrick Lewis, Robin Jia, Akari Asai, Danish Contractor, Douwe Kiela, Éric Gautier, Izacard Huan, Chen Sun, Jing Liu, Jinhyuk Lee, Jonathan H. Clark, Jonathan Herzig, Kai Sun, Kenton Lee, Kevin Lin, Matt Gardner, Matthew Lamm, Max Bartolo, Mor Geva, Nan Duan, Nitish Gupta, Panupong Pasupat, Pedro Rodríguez, Peng Qi, Rajarshi Das, Rodrigo Nogueira, Shayne Longpre, Shuohang Wang, Thomas Kwiatkowski, Tong Wang, Tushar Khot, Xiaodong Liu, Xinya Du, Yichen Jiang, Yizhong Wang, Reut Tsarfaty, Yiming Cui, Marco Ribeiro, Revanth Gangi Reddy, Bhavani Iyer, Md Arafat Sultan, Rong Zhang, Avirup Sil, Vittorio Castelli, Radu Florian, Patrick Ng, Ben Athiwaratkun, Elior Sulem, Jamaal Hay, Dan Roth, ; Bowei Zou, Zhifeng Li, Ai, Xiao-Yu Guo, Yuan-Fang Li, Gholamreza Haffari, Matthew Gardner 0001Openalex Unknown 2021 · DOI · OpenAlex

Introduction

The 3rd Workshop on Machine Reading for Question Answering (MRQA), held at EMNLP 2021, represents a significant milestone in the evolution of natural language understanding. While previous iterations of the workshop focused heavily on achieving state-of-the-art accuracy on English-centric benchmarks, the 2021 proceedings signal a strategic pivot. The field is moving beyond simple span-extraction tasks toward addressing the complex challenges of multilinguality, model interpretability, and real-world reliability.

This collection of research, comprising 16 archival papers and 23 non-archival works, defines the current frontier of how computational systems parse, retrieve, and generate answers from textual data.

Core Challenges: The English Bias and the Black Box

Despite the high performance of modern Question Answering (QA) systems on leaderboards, the workshop organizers identified two critical "desiderata" that remain largely unfulfilled by current architectures:

  1. The English Bias: The vast majority of QA research and evaluation is conducted on English datasets. This creates a performance gap for typologically diverse languages and limits the global utility of these systems.
  2. The Interpretability Problem: Modern large-scale neural models often function as "black boxes." They may produce correct answers by exploiting dataset shortcuts—such as simple lexical overlap between the question and the context—rather than through genuine reasoning. Without interpretability, these systems remain difficult to trust in high-stakes environments.

The technical contributions of the workshop reflect a diversified approach to solving these core challenges. Several key trends emerged from the archival papers:

  • Cross-Lingual Transfer: Researchers are increasingly utilizing bilingual alignment and sophisticated pre-training techniques to transfer QA capabilities from high-resource languages like English to lower-resource languages.
  • Generative vs. Extractive Paradigms: There is a notable shift from extractive QA (identifying a specific text span) to generative QA (synthesizing natural language answers). This shift necessitates a rethinking of training objectives and evaluation metrics.
  • Retrieval-Augmented Generation (RAG): Enhancements to models like REALM focus on making the retrieval of external knowledge more efficient and accurate, allowing models to ground their answers in broader contexts.
  • Debiasing and Robustness: To combat the tendency of models to "cheat" using word-matching shortcuts, new techniques involve using automated question generation to create more rigorous training sets that force deeper semantic understanding.
  • Multi-modal Reasoning: The proceedings also explored techniques for retrieving and reasoning across mixed data formats, such as combining structured tables with unstructured text.

Expanding the Benchmark Landscape

A primary contribution of the 3rd MRQA workshop is the introduction of several high-quality datasets designed to push the boundaries of the field:

  • MFAQ: A massive multilingual FAQ dataset, which received the Best Paper award for its role in improving global QA accessibility.
  • GermanQuAD & GermanDPR: These provide high-fidelity benchmarks for German-language QA and passage retrieval.
  • ParaShoot: A novel dataset specifically for Hebrew QA.
  • GANDALF: A specialized dataset focusing on character name descriptions within long-form fiction, testing a model's ability to handle narrative complexity.
  • Clinical Questionnaire Filling: A domain-specific dataset derived from human-machine interactions in the medical field, highlighting the nuances of professional vocabularies.

Key Insights and Real-World Utility

The workshop emphasized that accuracy is no longer the sole metric of success. A recurring theme was the "practice gap"—the friction between academic performance and real-world application. For instance, research into biomedical QA systems revealed that high-stakes domains require models to handle "knowledge conflicts," where a model's internal pre-trained memory might contradict the specific text it is currently analyzing.

Furthermore, the community is beginning to favor "Semantic Answer Similarity" over traditional exact-match or F1 scores. This acknowledges that a correct answer can be phrased in multiple ways, and evaluation metrics must be flexible enough to recognize semantic equivalence.

Conclusion and Future Directions

The 3rd MRQA Workshop successfully framed interpretability and multilinguality as requirements for deployment rather than mere research curiosities. However, significant bottlenecks remain. Interpreting the decision-making process of billion-parameter models is still an immense challenge, and the field is only in the early stages of truly integrated reasoning across multi-modal data formats. As the field moves forward, the focus will likely remain on bridging the gap between leaderboard success and the reliable, transparent performance required for professional and global applications.

Summary updated

BibTeX
@inproceedings{fisch2021proceedings,
  author = {Adam Fisch and Alon Talmor and Patrick Lewis and Robin Jia and Akari Asai and Danish Contractor and Douwe Kiela and Éric Gautier and Izacard Huan and Chen Sun and Jing Liu and Jinhyuk Lee and Jonathan H. Clark and Jonathan Herzig and Kai Sun and Kenton Lee and Kevin Lin and Matt Gardner and Matthew Lamm and Max Bartolo and Mor Geva and Nan Duan and Nitish Gupta and Panupong Pasupat and Pedro Rodríguez and Peng Qi and Rajarshi Das and Rodrigo Nogueira and Shayne Longpre and Shuohang Wang and Thomas Kwiatkowski and Tong Wang and Tushar Khot and Xiaodong Liu and Xinya Du and Yichen Jiang and Yizhong Wang and Reut Tsarfaty and Yiming Cui and Marco Ribeiro and Marco Ribeiro and Revanth Gangi Reddy and Bhavani Iyer and Md Arafat Sultan and Rong Zhang and Avirup Sil and Vittorio Castelli and Radu Florian and Matt Gardner and Patrick Ng and Ben Athiwaratkun and Elior Sulem and Jamaal Hay and Dan Roth and ; Bowei Zou and Zhifeng Li and Ai and Xiao-Yu Guo and Yuan-Fang Li and Gholamreza Haffari and Matthew Gardner},
  title = {{Proceedings of the 3rd Workshop on Machine Reading for Question Answering}},
  booktitle = {Openalex Unknown},
  year = {2021},
  doi = {10.18653/v1/2021.mrqa-1},
}
Figure