Abstract
Few-shot question answering (QA) aims at precisely discovering answers to a set of questions from context passages while only a few training samples are available. Although existing studies have made some progress and can usually achieve proper results, they suffer from understanding deep semantics for reasoning out the questions. In this paper, we develop Gotta, a Generative prOmpT-based daTa Augmentation framework to mitigate the challenge above. Inspired by the human reasoning process, we propose to integrate the cloze task to enhance few-shot QA learning. Following the recent success of prompt-tuning, we present the cloze task in the same format as the main QA task, allowing the model to learn both tasks seamlessly together to fully take advantage of the power of prompt-tuning. Extensive experiments on widely used benchmarks demonstrate that Gotta consistently outperforms competitive baselines, validating the effectiveness of our proposed prompt-tuning-based cloze task, which not only fine-tunes language models but also learns to guide reasoning in QA tasks. Further analysis shows that the prompt-based loss incorporates the auxiliary task better than the multi-task loss, highlighting the strength of prompt-tuning on the few-shot QA task.
Introduction
In Natural Language Processing, Question Answering (QA) is a fundamental task. While modern Large Language Models (LLMs) excel at QA when provided with extensive training data, they often struggle in few-shot settings—where only 16 to 128 examples are available. The core challenge in these scenarios is that standard fine-tuning often leads to "surface-level" mapping. Models learn to identify superficial patterns between questions and text but fail to digest the underlying entities and their relationships. To solve complex questions, a model must understand context as a human would: by identifying key entities and how they interact.
The Gotta Framework
The Gotta framework (Generative prOmpT-based daTa Augmentation) bridges this semantic gap by teaching models to "reason" through context using entity-aware cloze tasks. The framework consists of three main components:
- Entity-aware Text Masking: Unlike random word masking, Gotta utilizes the Wikidata knowledge base to identify critical entities (people, places, organizations) within the context. These specific entities are masked to create "cloze" (fill-in-the-blank) problems.
- Prompt-style Data Augmentation: To ensure the model treats auxiliary cloze tasks with the same importance as the primary QA task, Gotta formats them identically. An augmented example follows the structure:
Question: What is the masked entity? Answer: <masked_entity> Context: [Text with <mask>]. - Generative Prompt-Tuning: Utilizing a BART (encoder-decoder) architecture, the model trains on both original QA pairs and augmented cloze tasks simultaneously. This unified representation allows the model to leverage the power of prompt-tuning for more accurate reasoning.
Experimental Evaluation
The researchers evaluated Gotta using the MRQA 2019 shared task suite, encompassing eight diverse benchmark datasets including general news (SQuAD, NewsQA, NQ), web-scale data (TriviaQA, SearchQA), complex multi-hop questions (HotpotQA), and domain-specific texts (BioASQ, TextbookQA). Testing was conducted across four few-shot scenarios: 16, 32, 64, and 128 training examples.
Key Results and Insights
Gotta consistently outperformed competitive baselines like Splinter and FewshotQA across nearly all datasets and shot counts.
- Significant Performance Gains: On the TriviaQA dataset with only 16 examples, Gotta achieved a 34.3% relative improvement over the best baseline. It also showed strong performance on specialized data like TextbookQA, where understanding specific entities is crucial.
- Entities as Anchors of Reasoning: The results validate that masking factual entities forces the model to understand relationships within the text rather than just learning grammar from random masks.
- Format Consistency: Treating the auxiliary task as a prompt (Gotta) proved superior to traditional Multi-Task Learning (Gotta-MTL). Keeping the format identical allows for more seamless knowledge transfer between the cloze and QA tasks.
- Stability in Scarcity: Gotta demonstrated significantly lower variance than baselines, suggesting it produces more robust and reliable models in data-scarce environments.
Conclusion and Future Work
While Gotta is powerful, its benefits can vary based on context length; on datasets like SearchQA with extremely long contexts, the entity-matching process becomes more challenging. Future work may involve designing pre-training objectives more closely aligned with prompt-style cloze tasks and exploring how the framework scales with even larger backbone models beyond BART-large.
Summary updated