Single-dataset Experts for Multi-dataset Question Answering

Dan Friedman, Ben Dodge, Danqi Chen

Abstract: Dan Friedman, Ben Dodge, Danqi Chen. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing.

2021.

Introduction

In Reading Comprehension (RC), researchers often grapple with dozens of datasets like SQuAD, TriviaQA, and Natural Questions. Each dataset possesses unique characteristics, domains, and question styles. The conventional approach—training a single large model on a mixture of all available data—frequently encounters two significant hurdles: task interference, where datasets "fight" during training, and brittle generalization, where the resulting "average" model fails to adapt to specific new domains.

The authors propose MADE (Multi-Adapter Dataset Experts), a modular framework designed to capture universal linguistic regularities while preserving specialized domain expertise.

Methods: The MADE Framework

MADE utilizes Adapters—lightweight, trainable modules comprising roughly 1% of the total parameters—inserted between the layers of a pre-trained Transformer backbone (e.g., RoBERTa).

The MADE architecture features a shared backbone with dataset-specific adapters.

The training process involves two distinct phases:

  1. Joint Optimization: The shared backbone and all dataset-specific adapters are trained simultaneously across all source datasets. This allows the backbone to learn "universal" QA features.
  2. Adapter Tuning: The backbone is frozen, and each adapter is refined exclusively on its target dataset. This step captures the "special sauce" of specific domains without cross-task interference.

For transfer learning to unseen domains, MADE employs Parameter Averaging. Instead of selecting a single expert, a new adapter is initialized by averaging the weights of all source experts. This "average expert" provides a robust starting point for zero-shot or few-shot tasks.

Key Results

MADE demonstrates superior performance both in-distribution and in transfer scenarios.

In-Distribution Performance

On the six source datasets from the MRQA 2019 Shared Task, MADE (with adapter tuning) achieved an average 82.2 F1 score, surpassing standard multi-dataset fine-tuning (81.6 F1) and single-dataset fine-tuning (80.5 F1).

Zero-Shot and Few-Shot Transfer

The framework's modularity shines in transfer learning. When tested on unseen datasets (e.g., BioASQ, RACE) without any additional training, the MADE ensemble significantly outperformed established baselines.

Zero-shot F1 performance on target datasets comparing MADE variants to UnifiedQA and single-dataset adapters.

As shown in the results, the MADE ensemble reached 63.4 F1, a substantial improvement over UnifiedQA-base (50.9 F1). In few-shot settings with only 16 examples, MADE maintained its lead, achieving 63.5 F1 compared to 50.6 F1 for UnifiedQA.

Key Insights

  • Modular Expertise: MADE proves that specialized domain knowledge and general linguistic capability can coexist without compromise.
  • Weight Space Interpolation: The success of parameter averaging suggests that adapters learn compatible solutions, allowing their knowledge to be blended mathematically.
  • Low-Data Stability: MADE-averaged initializations provide a stable foundation for few-shot learning, reducing the volatility typically seen when fine-tuning on very small datasets.

Limitations and Future Work

Despite its strengths, MADE faces challenges with long-range reasoning (e.g., DuoRC) and the computational cost of running ensembles at inference time. Future research may explore dynamic "routing" of inputs to specific adapters or more advanced weight interpolation techniques to further refine domain transfer.

Summary updated

BibTeX
@inproceedings{friedman2021single,
  author = {Dan Friedman and Ben Dodge and Danqi Chen},
  title = {{Single-dataset Experts for Multi-dataset Question Answering}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  url = {https://openreview.net/forum?id=Uqg3Hqi9g_k},
}
Figure