Single-dataset Experts for Multi-dataset Question Answering

Dan Friedman, Ben Dodge, Danqi Chen

TL;DR

This paper introduces Multi-Adapter Dataset Experts (MADE), a novel approach to multi-dataset question answering that combines the strengths of single-dataset specialization and multi-dataset generalization. MADE achieves state-of-the-art performance on in-domain accuracy and significantly improves zero-shot generalization and few-shot transfer learning compared to existing methods.

What Problem Does This Paper Address?

Training reading comprehension models on multiple datasets simultaneously is a common practice to improve performance and generalization. However, traditional multi-task learning approaches can lead to models that overfit or underfit specific datasets, potentially hindering their ability to transfer to new, unseen data. Specialized single-dataset models, while accurate on their training data, lack the broad generalization capabilities of multi-dataset models. This paper aims to bridge this gap by developing a model that excels on its training data while also being a strong foundation for transfer learning to new datasets.

Methods

The core of the MADE approach is to leverage "adapters," which are lightweight, task-specific modules inserted between the layers of a pre-trained Transformer model (like BERT). Instead of training a single model on all datasets or training separate models for each dataset, MADE trains a collection of dataset-specific adapters that share an underlying Transformer.

The training process involves two phases:

  1. Joint Optimization: All parameters (shared Transformer and dataset-specific adapters/classifiers) are trained jointly on the source datasets. This allows the shared Transformer to learn cross-dataset regularities.
  2. Adapter Tuning: The shared Transformer is frozen, and each dataset-specific adapter is then fine-tuned individually on its corresponding dataset. This step refines the specialized knowledge for each dataset.

For zero-shot generalization to a new dataset, MADE averages the parameters of the trained adapters. For few-shot transfer learning, they explore two methods: averaging adapter parameters before fine-tuning, or fine-tuning all adapters and then taking a weighted average.

Datasets

The experiments utilize datasets from the MRQA 2019 shared task. These are divided into:

  • Six in-domain datasets: SQuAD 1.1, HotpotQA, TriviaQA, NewsQA, SearchQA, and Natural Questions.
  • Six out-of-domain datasets: BioASQ, DROP, DuoRC, RACE, RelationExtraction, and TextbookQA.

The paper downsamples the in-domain datasets to 75K training and 1K development examples for experiments. For few-shot transfer learning, they create datasets with K training examples (K=16, 64, 256) by reserving 400 examples as a test set.

Key Insights and Ideas

  • Decomposition is Key: Explicitly modeling datasets as a mixture, rather than a single monolithic task, allows for better specialization and generalization.
  • Adapters as Experts: Dataset-specific adapters act as "experts" for their respective data distributions, while the shared Transformer learns common underlying patterns.
  • Two-Phase Training Synergy: Joint optimization captures broad cross-dataset knowledge, while subsequent adapter tuning refines this knowledge for individual datasets, leading to superior performance.
  • Parameter Averaging for Transfer: A surprisingly simple approach of averaging adapter parameters works remarkably well for zero-shot generalization, outperforming more complex methods.
  • Adapter Tuning's Importance: While joint optimization is crucial, the separate adapter tuning phase significantly boosts performance, especially on datasets where joint optimization might have converged prematurely.

Key Results

  • In-domain Performance: MADE outperforms both single-dataset fine-tuning and multi-dataset fine-tuning baselines, achieving higher average F1 scores. For example, MADE (with adapter tuning) achieved an average F1 of 82.2 on the in-domain datasets, compared to 81.4 for dynamic sampling and 81.6 for multi-dataset fine-tuning.
  • Zero-shot Generalization: MADE significantly improves zero-shot generalization. Averaging MADE adapters achieved an average F1 of 63.5 on out-of-domain datasets, compared to 50.9 for UnifiedQA-base and 42.7 for single-dataset adapters.
  • Transfer Learning: MADE demonstrates strong few-shot transfer learning capabilities, especially with smaller amounts of target data. For K=256 examples, MADE (post avg.) achieved an average F1 of 66.6, outperforming multi-dataset fine-tuning (66.2) and single-dataset adapters (57.7).

Limitations and Future Work

  • Limitations: The paper notes that while parameter averaging is effective, it might not be optimal for all scenarios. The performance gap between MADE and single-dataset adapters in zero-shot generalization is larger than in in-domain performance, suggesting room for further improvement in how cross-dataset knowledge is transferred.
  • Future Work: The authors plan to explore more explicit mixture-modeling approaches for even better zero-shot prediction and transfer learning. They also suggest that this framework offers a versatile starting point for building new question-answering systems.

Summary updated

BibTeX
@inproceedings{friedman2021single,
  author = {Dan Friedman and Ben Dodge and Danqi Chen},
  title = {{Single-dataset Experts for Multi-dataset Question Answering}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  url = {https://openreview.net/forum?id=Uqg3Hqi9g_k},
}
Figure