QAmeleon: Multilingual QA with Only 5 Examples

Priyanka Agrawal, Chris Alberti, Fantine Huot, Joshua Maynez, Ji Ma, Sebastian Ruder, Kuzman Ganchev, Dipanjan Das, Mirella Lapata

TL;DR

QAMELEON is a "data-efficient" framework that builds high-performing multilingual Question Answering (QA) systems using only five human-labeled examples per language. By leveraging a massive language model to generate synthetic training data, it bridges 60% of the performance gap between zero-shot transfer and expensive, large-scale human annotation.

What Problem Does This Paper Address?

Building QA systems for the world's thousands of languages is a massive "cold start" problem. While English has an abundance of high-quality datasets (like SQuAD), most other languages have little to none.

Currently, researchers rely on two main workarounds:

  1. Zero-shot transfer: Training a model on English and hoping it generalizes to other languages.
  2. Machine Translation (MT): Translating English datasets into target languages.

Both have flaws. Zero-shot transfer often fails to capture linguistic nuances, and MT-based training data is riddled with "translationese" artifacts and a heavy bias toward English-centric topics. QAMELEON aims to overcome these by using a tiny amount of "gold" human data (just 5 examples) to steer a large language model into generating high-quality, native-sounding synthetic data for training.

Methods

The core of QAMELEON is a process called Prompt Tuning (PT) for data synthesis. Instead of manually writing a text prompt (Prompt Engineering), the researchers use five human-annotated examples to learn a "soft prompt"—a small set of tunable parameters (50 tokens) prepended to the input of a massive Pre-trained Language Model (specifically, PaLM-540B).

The workflow follows a "Distillation" pattern:

  • Two-Stage Generation: Given an unlabeled Wikipedia passage, the model first generates a potential answer, then generates a question for that answer.
  • Multilingual Bridge: The prompt encourages the model to generate the question/answer in both English and the target language simultaneously. This acts as a "chain-of-thought" bridge, improving the quality of the low-resource language output.
  • Fine-tuning: The resulting synthetic dataset (thousands of QA pairs) is used to fine-tune a smaller, more deployment-friendly model (mT5-XL).

This approach effectively uses the giant PaLM model as a "teacher" to create a custom training set for a smaller "student" model.

Datasets

The authors evaluated their approach on two major multilingual benchmarks:

  • TyDiQA-GoldP: A dataset covering 10 typologically diverse languages (e.g., Arabic, Bengali, Finnish, Telugu).
  • MLQA: A benchmark for 6 languages used primarily for cross-lingual evaluation.
  • Unlabeled Data: Passages were sampled from Wikipedia in the respective target languages to serve as the context for synthetic question generation.

Key Insights and Ideas

  • Distillation Beats Direct Prompting: Interestingly, fine-tuning a smaller model on QAMELEON’s synthetic data often outperforms using the giant 540B model directly in a few-shot setting. This suggests that data synthesis is a more efficient way to "extract" knowledge from LLMs for specific tasks.
  • The Power of Hybrid Data: The best results came from combining QAMELEON-generated data with MT-translated data. The MT data provides English-centric breadth, while QAMELEON provides language-specific depth and diversity, creating a more robust training signal.
  • BLEU is a Poor Metric for QA Quality: The authors found that while some models produced questions with higher BLEU scores (meaning they looked more like human questions), they weren't necessarily better for training. QAMELEON’s questions were more effective for the downstream QA task despite having slightly lower similarity to human-written text.
  • Data Efficiency: The system achieves significant gains with just 5 examples. While increasing to 50 or 100 examples helps, the "bang for your buck" is highest at the extreme low-resource end.

Key Results

  • Superiority over Baselines: On TyDiQA-GoldP, QAMELEON (5-shot) achieved a 70.2% Exact Match (EM) score, significantly outperforming the English-only baseline (58.5%) and the strong "translate-train" baseline (66.1%).
  • Bridging the Gap: The method closed nearly 60% of the performance gap between a zero-shot model and a fully supervised model trained on 50,000 human-labeled examples.
  • Cross-Dataset Generalization: The results held up on the MLQA benchmark, proving that the synthetic data generation scales across different languages and dataset styles.

Limitations and Future Work

  • Computational Cost: While the human effort is low (5 examples), the computational cost is high. Prompt-tuning a 540B parameter model requires significant hardware (the authors used 256 TPU v3 chips for 12 hours).
  • Model Dependency: The quality of the synthetic data is fundamentally tied to the capabilities of the underlying LLM. As LLMs improve, QAMELEON will likely become even more effective.
  • Saturation: The authors noted a saturation effect where generating more than ~1,000 QA pairs per language yielded diminishing returns.
  • Future Directions: The team suggests extending this "few-shot synthesis" approach to other complex multilingual tasks like document retrieval, summarization, and semantic parsing.

Summary updated

Abstract

The availability of large, high-quality datasets has been one of the main drivers of recent progress in question answering (QA). Such annotated datasets however are difficult and costly to collect, and rarely exist in languages other than English, rendering QA technology inaccessible to underrepresented languages. An alternative to building large monolingual training datasets is to leverage pre-trained language models (PLMs) under a few-shot learning setting. Our approach, QAmeleon, uses a PLM to automatically generate multilingual data upon which QA models are trained, thus avoiding costly annotation. Prompt tuning the PLM for data synthesis with only five examples per language delivers accuracy superior to translation-based baselines, bridges nearly 60% of the gap between an English-only baseline and a fully supervised upper bound trained on almost 50,000 hand labeled examples, and always leads to substantial improvements compared to fine-tuning a QA model directly on labeled examples in low resource settings. Experiments on the TyDiQA-GoldP and MLQA benchmarks show that few-shot prompt tuning for data synthesis scales across languages and is a viable alternative to large-scale annotation.
BibTeX
@article{agrawal2023qameleon,
  author = {Priyanka Agrawal and Chris Alberti and Fantine Huot and Joshua Maynez and Ji Ma and Sebastian Ruder and Kuzman Ganchev and Dipanjan Das and Mirella Lapata},
  title = {{QAmeleon: Multilingual QA with Only 5 Examples}},
  journal = {Trans. Assoc. Comput. Linguistics},
  year = {2023},
  doi = {10.1162/tacl\_a\_00625},
}

Loading related content...

Figure