FewshotQA: A simple framework for few-shot learning of question answering tasks using pre-trained text-to-text models

Rakesh Chada, Pradeep Natarajan

Introduction

Modern language models such as BERT, RoBERTa, and BART have achieved state-of-the-art performance on question answering (QA) benchmarks. However, these models typically require extensive fine-tuning on large-scale datasets like SQuAD, which contains over 100,000 annotated examples. In many real-world scenarios, such abundant labeled data is unavailable. When restricted to a "few-shot" setting (e.g., fewer than 100 examples), the performance of standard fine-tuned models often degrades significantly.

The authors of FewshotQA identify objective mismatch as a primary cause for this failure. Standard fine-tuning procedures introduce a new classification "head" to predict answer span boundaries—a task entirely absent from the model's original pre-training phase. Consequently, the model must learn a new logic from scratch using a very limited signal.

The FewshotQA Framework

FewshotQA addresses this mismatch by aligning the fine-tuning task with the model's original pre-training objective. Instead of modifying the model architecture, the framework reformulates the QA task as a text-to-text "denoising" problem, similar to the tasks solved during pre-training.

Comparison of pre-training objectives for BERT*, BART, and T5 models.

Alignment Strategy

The framework utilizes text-to-text architectures like BART and T5. The input is formatted using a specific prompt template: Question: [q] Answer: <mask> Context: [c].

  • BART: The model is fine-tuned to reconstruct the entire input string, filling the <mask> with the correct answer, matching its pre-training objective of input reconstruction.
  • T5: The model is trained to generate only the missing answer span, matching its masked span generation pre-training task.

By using the model exactly as it was pre-trained, FewshotQA eliminates the need for new parameters, allowing the model to leverage its latent linguistic knowledge immediately.

Fine-tuning objectives for QA: Standard classification-based approaches (BERT*, BART) vs. the generative approach (T5).

Experimental Evaluation

The framework was evaluated across several diverse benchmarks, including the MRQA Shared Task (8 datasets) and TydiQA (a multilingual benchmark covering 9 languages).

Key Results

The results demonstrate that alignment is a powerful tool for low-data regimes:

  • Performance Gains: FewshotBART outperformed standard BART fine-tuning by an average of 34.2 F1 points in 16-shot settings.
  • SQuAD Benchmark: With only 32 examples, FewshotBART-large achieved a 72.3 F1 score, vastly outperforming standard RoBERTa-base (7.7 F1 with 16 examples).
  • Multilingual Success: On TydiQA, FewshotmBART surpassed the XLM-Roberta-large baseline by an average of 33 F1 points in low-data scenarios.

Discussion

A critical finding of the study is the "Context Last" strategy. Placing the context at the end of the input string allows the decoder to generate the answer more efficiently without auto-regressively processing long context sequences first. Furthermore, the framework benefits significantly from model scaling; larger versions of BART and T5 showed more pronounced improvements, suggesting that alignment helps unlock deeper pre-trained knowledge.

Conclusion

FewshotQA provides a simple yet effective method for few-shot QA by minimizing the gap between pre-training and fine-tuning. While currently focused on extractive QA, the dramatic gains in data efficiency make FewshotQA a compelling choice for low-resource domains where traditional fine-tuning fails.

Summary updated

Abstract

The task of learning from only a few examples (called a few-shot setting) is of key importance and relevance to a real-world setting. For question answering (QA), the current state-of-the-art pre-trained models typically need fine-tuning on tens of thousands of examples to obtain good results. Their performance degrades significantly in a few-shot setting (< 100 examples). To address this, we propose a simple fine-tuning framework that leverages pre-trained text-to-text models and is directly aligned with their pre-training framework. Specifically, we construct the input as a concatenation of the question, a mask token representing the answer span and a context. Given this input, the model is fine-tuned using the same objective as that of its pre-training objective. Through experimental studies on various few-shot configurations, we show that this formulation leads to significant gains on multiple QA benchmarks (an absolute gain of 34.2 F1 points on average when there are only 16 training examples). The gains extend further when used with larger models (Eg:- 72.3 F1 on SQuAD using BART-large with only 32 examples) and translate well to a multilingual setting . On the multilingual TydiQA benchmark, our model outperforms the XLM-Roberta-large by an absolute margin of upto 40 F1 points and an average of 33 F1 points in a few-shot setting (<= 64 training examples). We conduct detailed ablation studies to analyze factors contributing to these gains.
BibTeX
@inproceedings{chada2021fewshotqa,
  author = {Rakesh Chada and Pradeep Natarajan},
  title = {{FewshotQA: A simple framework for few-shot learning of question answering tasks using pre-trained text-to-text models}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  doi = {10.18653/v1/2021.emnlp-main.491},
}
Figure