QAmeleon: Multilingual QA with Only 5 Examples

Priyanka Agrawal, Chris Alberti, Fantine Huot, Joshua Maynez, Ji Ma, Sebastian Ruder, Kuzman Ganchev, Dipanjan Das, Mirella Lapata

Abstract

The availability of large, high-quality datasets has been one of the main drivers of recent progress in question answering (QA). Such annotated datasets however are difficult and costly to collect, and rarely exist in languages other than English, rendering QA technology inaccessible to underrepresented languages. An alternative to building large monolingual training datasets is to leverage pre-trained language models (PLMs) under a few-shot learning setting. Our approach, QAmeleon, uses a PLM to automatically generate multilingual data upon which QA models are trained, thus avoiding costly annotation. Prompt tuning the PLM for data synthesis with only five examples per language delivers accuracy superior to translation-based baselines, bridges nearly 60% of the gap between an English-only baseline and a fully supervised upper bound trained on almost 50,000 hand labeled examples, and always leads to substantial improvements compared to fine-tuning a QA model directly on labeled examples in low resource settings. Experiments on the TyDiQA-GoldP and MLQA benchmarks show that few-shot prompt tuning for data synthesis scales across languages and is a viable alternative to large-scale annotation.

Introduction

Building Question Answering (QA) systems for the world's thousands of languages presents a significant "cold start" problem. While English benefits from an abundance of high-quality datasets like SQuAD, most other languages suffer from a severe lack of annotated data. Current workarounds typically involve zero-shot transfer (training on English and testing on target languages) or Machine Translation (MT) to translate English datasets. However, zero-shot transfer often misses linguistic nuances, and MT-based data frequently contains "translationese" artifacts and English-centric biases.

QAmeleon addresses these challenges by using a "data-efficient" framework that requires only five human-labeled examples per language. It leverages the generative power of large language models (LLMs) to create high-quality, native-sounding synthetic training data, effectively bridging the gap between zero-shot transfer and expensive large-scale human annotation.

Methodology: Prompt Tuning for Data Synthesis

The core innovation of QAmeleon is Prompt Tuning (PT) for data synthesis. Rather than manual prompt engineering, the system uses five human-annotated examples to learn a "soft prompt"—a set of 50 tunable parameters prepended to the input of a massive Pre-trained Language Model (specifically, PaLM-540B).

The workflow follows a distillation pattern:

  1. Two-Stage Generation: Given an unlabeled Wikipedia passage, the model first generates a potential answer and then a corresponding question.
  2. Multilingual Bridge: The prompt encourages simultaneous generation in both English and the target language, acting as a "chain-of-thought" bridge to improve output quality in low-resource languages.
  3. Fine-tuning: The resulting synthetic dataset is used to fine-tune a smaller, more efficient "student" model (mT5-XL) for deployment.
Comparison of QA training methodologies: English-only, Machine Translation, and the QAmeleon few-shot approach.

Experimental Setup

The approach was evaluated on two major multilingual benchmarks:

  • TyDiQA-GoldP: Covering 10 typologically diverse languages including Arabic, Bengali, Finnish, and Telugu.
  • MLQA: A benchmark for 6 languages used for cross-lingual evaluation.
  • Unlabeled Data: Passages were sampled from Wikipedia in the target languages to serve as context for synthetic question generation.

Key Results

QAmeleon demonstrates significant performance gains over standard baselines. On the TyDiQA-GoldP benchmark, the 5-shot QAmeleon approach achieved a 70.2% Exact Match (EM) score. This significantly outperforms the English-only baseline (58.5%) and the strong "translate-train" baseline (66.1%).

Exact Match (EM) scores on the Multilingual QA task, showing QAmeleon's superior performance compared to English-only and MT-based baselines.

Key insights from the evaluation include:

  • Bridging the Gap: The method closed nearly 60% of the performance gap between zero-shot models and fully supervised models trained on 50,000 human-labeled examples.
  • Distillation Efficiency: Fine-tuning a smaller model on synthetic data often outperformed using the massive 540B model directly in a few-shot setting, suggesting data synthesis is a more efficient way to extract LLM knowledge.
  • Hybrid Data Benefits: Combining QAmeleon-generated data with MT-translated data yielded the best results, balancing English-centric breadth with language-specific depth.

Conclusion and Future Work

While QAmeleon requires high initial computational resources for prompt-tuning the 540B parameter model, it drastically reduces the human effort required for multilingual QA. The system achieves high data efficiency, with the most significant gains occurring at the extreme low-resource end (5 examples). Future work aims to extend this few-shot synthesis approach to other complex tasks such as document retrieval, summarization, and semantic parsing.

Summary updated

Abstract

The availability of large, high-quality datasets has been one of the main drivers of recent progress in question answering (QA). Such annotated datasets however are difficult and costly to collect, and rarely exist in languages other than English, rendering QA technology inaccessible to underrepresented languages. An alternative to building large monolingual training datasets is to leverage pre-trained language models (PLMs) under a few-shot learning setting. Our approach, QAmeleon, uses a PLM to automatically generate multilingual data upon which QA models are trained, thus avoiding costly annotation. Prompt tuning the PLM for data synthesis with only five examples per language delivers accuracy superior to translation-based baselines, bridges nearly 60% of the gap between an English-only baseline and a fully supervised upper bound trained on almost 50,000 hand labeled examples, and always leads to substantial improvements compared to fine-tuning a QA model directly on labeled examples in low resource settings. Experiments on the TyDiQA-GoldP and MLQA benchmarks show that few-shot prompt tuning for data synthesis scales across languages and is a viable alternative to large-scale annotation.
BibTeX
@article{agrawal2023qameleon,
  author = {Priyanka Agrawal and Chris Alberti and Fantine Huot and Joshua Maynez and Ji Ma and Sebastian Ruder and Kuzman Ganchev and Dipanjan Das and Mirella Lapata},
  title = {{QAmeleon: Multilingual QA with Only 5 Examples}},
  journal = {Trans. Assoc. Comput. Linguistics},
  year = {2023},
  doi = {10.1162/tacl\_a\_00625},
}
Figure