General-Purpose Question-Answering with Macaw

Oyvind Tafjord, Peter ClarkarXiv 2021 · OpenReview

Abstract

Despite the successes of pretrained language models, there are still few high-quality, general-purpose QA systems that are freely available. In response, we present Macaw, a versatile, generative question-answering (QA) system that we are making available to the community. Macaw is built on UnifiedQA, itself built on T5, and exhibits strong performance, zero-shot, on a wide variety of topics, including outperforming GPT-3 by over 10% (absolute) on Challenge300, a suite of 300 challenge questions, despite being an order of magnitude smaller (11 billion vs. 175 billion parameters). In addition, Macaw allows different permutations ("angles") of its inputs and outputs to be used. Macaw is freely available at https://github.com/allenai/macaw.

Introduction

While Large Language Models (LLMs) have shown incredible promise, most high-quality question-answering systems are either specialized for narrow tasks (like selecting a text span) or are locked behind proprietary APIs. The community lacks a flexible, "off-the-shelf" generative QA system that can handle diverse inputs and outputs. Standard models are often "one-way streets"—taking a question and providing an answer. Macaw addresses this gap by modeling the holistic relationship between questions, answers, contexts, and explanations, allowing it to function in any direction.

Multi-Angle Methodology

Macaw is built on the backbone of T5 (Text-to-Text Transfer Transformer) and UnifiedQA. Its primary innovation is Multi-Angle Training. Instead of a fixed mapping, the model is trained on various permutations of five "slots": Question (Q), Answer (A), Context (C), Multiple-choice options (M), and Explanation (E).

Definitions of the multi-angle input/output permutations used to train Macaw.

By shuffling these slots during training, Macaw learns to generate answers from questions, questions from answers, or even plausible distractor options for multiple-choice formats. This approach allows the model to learn a more robust representation of knowledge and reasoning than traditional single-task training. For instance, the model can generate an explanation for why an answer is correct (QA → E) or generate a question given an answer and context (AC → QM).

Evaluation and Results

The model underwent a two-stage training process involving seven core datasets (including SQuAD 2.0, ARC, and RACE) followed by fine-tuning on science questions and explanations from the WorldTree V2 bank. To test its limits, the authors introduced Challenge300, a benchmark of 300 difficult questions across 22 categories designed to probe reasoning beyond simple retrieval.

Performance on Challenge300

Macaw-11B achieved a score of 75%, significantly outperforming much larger models like GPT-3 (64.9%) and Jurassic-1 (64.9%). Notably, Macaw produced only 2 "incoherent" answers out of 300, compared to 10 from GPT-3 and 28 from T5-CBQA.

Comparative performance of Macaw against other models across different question categories in the Challenge300 benchmark.

As shown in the performance breakdown, Macaw excels in categories like meta-reasoning and general knowledge. Its efficiency—beating a model 16 times its size—suggests that multi-angle training is a powerful alternative to sheer parameter scaling for QA tasks. Furthermore, the paper finds that including an explanation in the input (the QMEC → A angle) significantly improves answer accuracy.

Discussion and Limitations

Macaw demonstrates remarkable zero-shot versatility, such as answering complex "what if" scenarios. However, the Challenge300 evaluation also highlights clear "breaking points" where the model still struggles:

  • Arithmetic: It fails at multi-digit math operations.
  • Entity Tracking: It struggles with "state change" problems (e.g., tracking an object's location after it is moved).
  • Spatial Reasoning: It often confuses relational descriptions like "left" and "right" in complex scenarios.
  • False Presuppositions: While it can catch some tricks (e.g., "When did Tom Hanks land on the moon?"), it is still frequently fooled by questions that assume a false premise.

Future work will focus on better integrating formal reasoning or latent spatial layouts to overcome these specific hurdles. Macaw remains an open-source tool for the community to build upon.

Summary updated

Abstract

Despite the successes of pretrained language models, there are still few high-quality, general-purpose QA systems that are freely available. In response, we present Macaw, a versatile, generative question-answering (QA) system that we are making available to the community. Macaw is built on UnifiedQA, itself built on T5, and exhibits strong performance, zero-shot, on a wide variety of topics, including outperforming GPT-3 by over 10% (absolute) on Challenge300, a suite of 300 challenge questions, despite being an order of magnitude smaller (11 billion vs. 175 billion parameters). In addition, Macaw allows different permutations ("angles") of its inputs and outputs to be used, for example Macaw can take a question and produce an answer; or take an answer and produce a question; or take an answer and question, and produce multiple-choice options. We describe the system, and illustrate a variety of question types where it produces surprisingly good answers, well outside the training setup. We also identify question classes where it still appears to struggle, offering insights into the limitations of pretrained language models. Macaw is freely available, and we hope that it proves useful to the community. Macaw is available at https://github.com/allenai/macaw
BibTeX
@misc{tafjord2021general,
  author = {Oyvind Tafjord and Peter Clark},
  title = {{General-Purpose Question-Answering with Macaw}},
  year = {2021},
  eprint = {2109.02593},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2109.02593},
}
Figure