TL;DR
GAAMA 2.0 is a state-of-the-art multilingual question-answering system that bridges the gap between finding specific facts and answering yes/no questions. By integrating specialized classifiers and a novel score normalization technique, it provides direct "YES/NO" answers alongside highlighted evidence, achieving top-tier performance on the TyDi QA leaderboard while offering a memory-efficient version using parameter-sharing adapters.
What Problem Does This Paper Address?
Most Machine Reading Comprehension (MRC) systems are specialists: they are either very good at extracting short "factoid" answers (e.g., "What is the capital of France?") or they are designed to handle "boolean" questions (e.g., "Is Paris the capital of France?").
However, real-world users need a generalist. Current datasets like TyDi QA mix these question types, but existing models often struggle to handle them in a unified way. Specifically, boolean questions require more than just a binary label; they need supporting evidence from the text to be trustworthy. Furthermore, deploying multiple large transformer models—one for each task—is computationally expensive and memory-intensive, making it difficult to run these systems in resource-constrained environments.
Methods
The authors propose an integrated pipeline, GAAMA 2.0, which consists of several key components:
- Question Type Classifier: A multilingual transformer (mBERT) that determines if a question is boolean or extractive.
- GAAMA (Span Extractor): The core engine, based on XLM-RoBERTa-large, which finds the most relevant text span. For factoid questions, this is the answer; for boolean questions, this serves as the "supporting evidence."
- Boolean Answer Classifier: A binary classifier that looks at the question and the document to decide on a "YES" or "NO" answer.
- Score Normalizer: Perhaps the most critical innovation. The authors found that boolean and extractive questions produce wildly different confidence score distributions (see Figure 4 below). The normalizer uses logistic regression to scale these scores to a common [0, 1] interval, allowing the system to accurately decide if a question is "unanswerable."
To address hardware constraints, the team also developed an Adapter-based version. Instead of deploying three separate transformer stacks, they use one base model and add tiny, task-specific "adapter" layers (less than 1% of the total parameters) for the classification tasks.
Datasets
The system was trained and evaluated using:
- TyDi QA: A multilingual benchmark covering 11 typologically diverse languages, featuring a mix of extractive and boolean questions.
- BoolQ-X: An enhanced version of the BoolQ dataset, modified to provide more context, making it more compatible with the MRC task.
- Natural Questions & MS Marco: Used as additional sources of mixed-type question data.
Key Insights and Ideas
- The "Confidence Gap": The authors discovered that boolean questions often receive lower confidence scores from standard models because the "answer span" (the whole paragraph) is much longer than typical factoid answers. Without the Score Normalizer, the system would incorrectly label most boolean questions as "unanswerable."
- Adapters are "Free" Efficiency: Moving to an adapter-based architecture provided a massive reduction in GPU memory footprint (about 3x) without sacrificing accuracy. In fact, the adapter-based question type classifier actually performed better than the standalone version.
- Evidence is Essential: For boolean questions, providing the "YES/NO" is only half the battle. By repurposing the span extractor to find supporting evidence, the system becomes significantly more interpretable for the end-user.
Key Results
GAAMA 2.0 achieved 1st place on the TyDi QA leaderboard at the time of publication.
- Accuracy Boost: The integrated system showed a clear gain of 4 F1 points over the baseline GAAMA model on the TyDi QA development set (see Table 2).
- Normalization Impact: As shown in the confusion matrix (Table 1), the score normalizer dramatically improved the handling of boolean questions, moving hundreds of correct "YES/NO" answers from "below threshold" (unanswerable) to "above threshold."
- Memory Efficiency: The adapter configuration reduced the model's memory footprint from 3204 MiB to 1074 MiB while slightly improving the F1 score from 72.6 to 73.0 (see Table 3).
Limitations and Future Work
The authors acknowledge a few areas for improvement:
- Training Data Gaps: The TyDi QA dataset does not provide labels for unanswerable boolean questions, which limited the training of the question type classifier.
- Linguistic Heuristics: The question type classifier currently relies heavily on linguistic cues (like "Is" vs. "Who"), which may not capture every nuance of how questions are phrased across all languages.
- Future Directions: The team plans to expand the system to handle even more complex answer types, such as "how-to" questions that require a list of steps rather than a single span or a binary choice.
Summary updated