Few-shot Unified Question Answering: Tuning Models or Prompts?

Srijan Bansal, Semih Yavuz, Bo Pang, Meghana Moorthy Bhat, Yingbo Zhou

Abstract

Question Answering (QA) remains a fragmented field in Natural Language Processing, with diverse datasets requiring specialized reasoning skills ranging from temporal logic to commonsense. While unified QA models aim to address this by handling multiple tasks simultaneously, they often require extensive data and significant computational resources for fine-tuning. This paper investigates the efficacy of Prompt Tuning versus Model Tuning in the few-shot regime (16 to 1024 examples). The researchers demonstrate that with strategic initialization from similar tasks, prompt tuning can match the performance of full model tuning while remaining significantly more parameter-efficient and scalable.

Introduction

In the landscape of modern NLP, Question Answering is a cornerstone capability. However, the domain is highly heterogeneous; a model proficient in reading comprehension may fail at logical reasoning or conversational context. Unified QA models represent an attempt to bridge these gaps, but they face a fundamental trade-off in low-resource settings: should one update the billions of parameters in a large language model (Model Tuning), or keep the model frozen and only optimize a small set of "soft prompt" vectors (Prompt Tuning)?

Model tuning is powerful but computationally expensive and prone to "catastrophic forgetting" of previous tasks. Prompt tuning offers a lightweight alternative but often struggles to converge when training data is scarce. This study evaluates these paradigms using the T5 (Text-to-Text Transfer Transformer) architecture to determine which approach best serves the needs of unified, few-shot QA.

Methodology

The researchers compared several distinct training paradigms:

  1. Model Tuning (FT & FT-MT): Standard fine-tuning of all model weights. This was tested on single tasks (FT) and across multiple tasks simultaneously (FT-MT).
  2. Vanilla Prompt Tuning (PT-R): Trainable soft prompt tokens are added to the input while the backbone model remains frozen. In this baseline, prompts are initialized randomly.
  3. Initialized Prompt Tuning (PT-F & PT-C): To overcome the limitations of random starts, prompts are initialized by averaging the best-performing prompts from "source" tasks. PT-F uses tasks sharing the same format, while PT-C selects tasks based on overall similarity across the collection.
  4. ATTEMPT: A state-of-the-art method that utilizes an attention mechanism to interpolate between multiple source prompts dynamically.

Experimental Setup

The study utilized a robust suite of 16 target datasets to evaluate generalization across various reasoning types:

  • Reading Comprehension: TweetQA, IIRC, and MCTest.
  • Commonsense and Reasoning: BoolQ, PIQA, SocialIQa, and COPA.
  • Specialized Logic: ReClor (logical reasoning) and ROPES (causal reasoning).

To facilitate prompt initialization, high-resource "source" datasets—including SQuAD, Natural Questions, and TriviaQA—were used to train initial prompt vectors, totaling over 30,000 training instances.

Key Insights

The Role of Initialization

The most critical finding was that initialization is the "secret sauce" for prompt tuning in low-resource settings. While random initialization (PT-R) performs poorly with only a few examples, "borrowing" a prompt from a similar task (PT-C) allows the model to jump-start its learning. This initialization allows prompt tuning to achieve parity with full model tuning in few-shot scenarios.

The Multi-Task Paradox

Sharing parameters across tasks (FT-MT) proved to be a significant advantage in few-shot scenarios, as tasks can provide mutual reinforcement. However, the researchers observed a paradox: in full-data scenarios, multi-tasking can lead to "negative interference," where unrelated tasks degrade each other's performance. Prompt tuning avoids some of this interference by keeping the core model weights frozen.

Simplicity and Inductive Bias

Surprisingly, simple heuristic initialization (averaging similar prompts) performed as effectively as complex, parameter-heavy transfer methods like ATTEMPT. Furthermore, prompt tuning appears to benefit more from the pre-trained backbone's inductive bias than model tuning does. The frozen model acts as a stable repository of knowledge, while the soft prompt serves as a specialized trigger for specific task logic.

Results

  • Competitive Performance: At the 16-shot level, Prompt Tuning with initialization (PT-C) achieved a score of 45.55, nearly identical to Multi-Task Model Tuning (45.62), despite training only a tiny fraction of the total parameters.
  • Pre-training Impact: Utilizing a backbone model pre-trained on diverse QA tasks (PrLM) improved prompt tuning performance by approximately 25% across all benchmarks.
  • Scaling Trends: While larger models (T5-Large vs. T5-Base) consistently improved model tuning, the gains for prompt tuning were more concentrated in the extreme low-data regime, with diminishing returns as the number of training examples increased.

Conclusion and Future Work

This research highlights prompt tuning as a viable, efficient path for unified QA, provided that proper initialization strategies are employed. However, challenges remain. Few-shot results exhibit high variance depending on the specific examples selected, and the performance gap between prompt and model tuning may widen as models scale to even larger sizes. Future work will need to address these stability issues and explore whether these prompt-transfer benefits extend to cross-lingual and multi-lingual QA environments.

Summary updated

Abstract

Question-answering (QA) tasks often investigate specific question types, knowledge domains, or reasoning skills, leading to specialized models catering to specific categories of QA tasks. While recent research has explored the idea of unified QA models, such models are usually explored for high-resource scenarios and require re-training to extend their capabilities. To overcome these drawbacks, the paper explores the potential of two paradigms of tuning, model, and prompts, for unified QA under a low-resource setting. The paper provides an exhaustive analysis of their applicability using 16 QA datasets, revealing that prompt tuning can perform as well as model tuning in a few-shot setting with a good initialization. The study also shows that parameter-sharing results in superior few-shot performance, simple knowledge transfer techniques for prompt initialization can be effective, and prompt tuning achieves a significant performance boost from pre-training in a low-resource regime. The research offers insights into the advantages and limitations of prompt tuning for unified QA in a few-shot setting, contributing to the development of effective and efficient systems in low-resource scenarios.
BibTeX
@inproceedings{bansal2023few,
  author = {Srijan Bansal and Semih Yavuz and Bo Pang and Meghana Moorthy Bhat and Yingbo Zhou},
  title = {{Few-shot Unified Question Answering: Tuning Models or Prompts?}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  doi = {10.18653/v1/2023.findings-emnlp.550},
}
Figure