Abstract
Despite the successes of pretrained language models, there are still few high-quality, general-purpose QA systems that are freely available. In response, we present Macaw, a versatile, generative question-answering (QA) system that we are making available to the community. Macaw is built on UnifiedQA, itself built on T5, and exhibits strong performance, zero-shot, on a wide variety of topics, including outperforming GPT-3 by over 10% (absolute) on Challenge300, a suite of 300 challenge questions, despite being an order of magnitude smaller (11 billion vs. 175 billion parameters). In addition, Macaw allows different permutations ("angles") of its inputs and outputs to be used. Macaw is freely available at https://github.com/allenai/macaw.
Introduction
While Large Language Models (LLMs) have shown incredible promise, most high-quality question-answering systems are either specialized for narrow tasks (like selecting a text span) or are locked behind proprietary APIs. The community lacks a flexible, "off-the-shelf" generative QA system that can handle diverse inputs and outputs. Standard models are often "one-way streets"—taking a question and providing an answer. Macaw addresses this gap by modeling the holistic relationship between questions, answers, contexts, and explanations, allowing it to function in any direction.
Multi-Angle Methodology
Macaw is built on the backbone of T5 (Text-to-Text Transfer Transformer) and UnifiedQA. Its primary innovation is Multi-Angle Training. Instead of a fixed mapping, the model is trained on various permutations of five "slots": Question (Q), Answer (A), Context (C), Multiple-choice options (M), and Explanation (E).
By shuffling these slots during training, Macaw learns to generate answers from questions, questions from answers, or even plausible distractor options for multiple-choice formats. This approach allows the model to learn a more robust representation of knowledge and reasoning than traditional single-task training. For instance, the model can generate an explanation for why an answer is correct (QA → E) or generate a question given an answer and context (AC → QM).
Evaluation and Results
The model underwent a two-stage training process involving seven core datasets (including SQuAD 2.0, ARC, and RACE) followed by fine-tuning on science questions and explanations from the WorldTree V2 bank. To test its limits, the authors introduced Challenge300, a benchmark of 300 difficult questions across 22 categories designed to probe reasoning beyond simple retrieval.
Performance on Challenge300
Macaw-11B achieved a score of 75%, significantly outperforming much larger models like GPT-3 (64.9%) and Jurassic-1 (64.9%). Notably, Macaw produced only 2 "incoherent" answers out of 300, compared to 10 from GPT-3 and 28 from T5-CBQA.
As shown in the performance breakdown, Macaw excels in categories like meta-reasoning and general knowledge. Its efficiency—beating a model 16 times its size—suggests that multi-angle training is a powerful alternative to sheer parameter scaling for QA tasks. Furthermore, the paper finds that including an explanation in the input (the QMEC → A angle) significantly improves answer accuracy.
Discussion and Limitations
Macaw demonstrates remarkable zero-shot versatility, such as answering complex "what if" scenarios. However, the Challenge300 evaluation also highlights clear "breaking points" where the model still struggles:
- Arithmetic: It fails at multi-digit math operations.
- Entity Tracking: It struggles with "state change" problems (e.g., tracking an object's location after it is moved).
- Spatial Reasoning: It often confuses relational descriptions like "left" and "right" in complex scenarios.
- False Presuppositions: While it can catch some tricks (e.g., "When did Tom Hanks land on the moon?"), it is still frequently fooled by questions that assume a false premise.
Future work will focus on better integrating formal reasoning or latent spatial layouts to overcome these specific hurdles. Macaw remains an open-source tool for the community to build upon.
Summary updated