TL;DR
GAAMA 2.0 is a state-of-the-art, multilingual machine reading comprehension (MRC) system that bridges the gap between extractive (factoid) and boolean (Yes/No) questions. By integrating specialized classifiers and a novel score normalization technique, it achieved the top spot on the TyDi QA leaderboard while offering a memory-efficient version powered by parameter-sharing adapters.
What Problem Does This Paper Address?
In the world of Machine Reading Comprehension (MRC), most models are specialized: they either excel at finding a specific text snippet to answer a "who/what/where" question (extractive) or they are trained to provide a simple "Yes/No" (boolean).
However, real-world users don't distinguish between these types, and boolean answers are often unsatisfying without context. If you ask, "Is the Mississippi the longest river?" and the system just says "No," you immediately want to know why. Current systems struggle to provide both the binary answer and the supporting evidence in a unified, multilingual framework. Furthermore, identifying "unanswerable" questions is difficult because confidence scores for boolean and extractive answers often exist on completely different scales, making a single "rejection threshold" ineffective.
Methods
The authors propose GAAMA 2.0, an integrated pipeline consisting of several key modules:
- The Core Extractor (GAAMA): Based on a 24-layer XLM-RoBERTa-large model, this component is trained via multi-teacher knowledge distillation to find answer spans in text.
- Question Type Classifier: A multilingual BERT (mBERT) model that determines if a question is boolean or extractive.
- Boolean Answer Classifier: If a question is flagged as boolean, this binary classifier determines the Yes/No status.
- Score Normalizer: This is a critical "glue" component. It uses logistic regression to scale the raw scores from different components into a unified [0, 1] range. This allows the system to use a single threshold to decide if a question is "unanswerable" regardless of its type.
- Efficiency via Adapters: To avoid the massive GPU memory footprint of running three separate transformer stacks, the authors implemented a version using adapters. They use the large span-extraction model as a "base" and add tiny, task-specific layers (less than 1% of the total parameters) for the other classifications.
Datasets
- TyDi QA: A benchmark for information-seeking question answering in typologically diverse languages. It contains a mix of extractive and boolean questions (about 10% boolean).
- BoolQ-X: An enhanced version of the BoolQ dataset, modified to provide more context, making it more compatible with MRC tasks.
- Natural Questions & MS Marco: Used for comparison and as part of the multi-teacher distillation process.
Key Insights and Ideas
- Evidence Matters for Boolean Questions: Unlike traditional boolean classifiers that just output a label, GAAMA 2.0 treats the span produced by the extractive model as "supporting evidence" for the Yes/No answer, providing a much more transparent user experience.
- The Normalization "Aha" Moment: The authors discovered that raw scores for boolean questions were significantly lower than for extractive ones. Without the Score Normalizer, the system would incorrectly label the majority of correct boolean answers as "unanswerable" because they fell below the global threshold.
- Adapters are a "Free Lunch": The adapter-based implementation performed nearly identically to (and in some cases better than) the version with independent model stacks, while using only one-third of the memory. This makes the system viable for deployment in resource-constrained environments.
Key Results
- Leaderboard Success: GAAMA 2.0 ranked first on the TyDi QA leaderboard at the time of publication.
- Performance Boost: The integrated approach yielded a 4-point F1 improvement over baseline models that didn't handle boolean questions specifically.
- High Accuracy: The question type classifier achieved an impressive 99.2% F1 score for boolean questions, showing that linguistic cues (like starting a sentence with "Is" or "Does") are highly reliable signals.
- Memory Efficiency: The adapter-based model reduced the memory footprint from 3204 MiB to 1074 MiB without sacrificing accuracy.
Limitations and Future Work
- Answer Diversity: While the system handles boolean and extractive questions, it does not yet handle "how-to" questions or queries that require a list of items as an answer.
- Evidence Selection: The "supporting evidence" for boolean questions is currently limited to the single best span found by the extractive model. Future iterations could explore more complex evidence synthesis or multi-sentence justifications.
- Language Expansion: While multilingual, the performance varies across languages depending on the availability of training data in the TyDi QA set.
Summary updated