UnitedQA: A Hybrid Approach for Open Domain Question Answering

Hao Cheng, Yelong Shen, Xiaodong Liu, Pengcheng He, Weizhu Chen, Jianfeng Gao

TL;DR

Most open-domain question answering (QA) systems choose between two reader paradigms: extractive (selecting text spans) or generative (decoding free-form text). UnitedQA demonstrates that these two families make surprisingly different predictions (less than 50% agreement) despite similar accuracies. By introducing tailored training improvements to both readers and combining them with a simple linear re-ranking ensemble, UnitedQA sets a new state-of-the-art on NaturalQuestions and TriviaQA.

What Problem Does This Paper Address?

In retrieval-augmented open-domain QA, an initial retrieval step surfaces candidate passages from a vast corpus (such as Wikipedia), after which a reader module infers the final answer. Research has historically split into two competing reader camps:

  1. Extractive readers (e.g., DPR, REALM) that identify contiguous token spans within retrieved contexts.
  2. Generative readers (e.g., T5-FID, RAG) that synthesize answers token by token.

Existing systems commit exclusively to one paradigm. The authors explore whether these architectures develop fundamentally distinct inference strategies, whether their errors are complementary, and how best to combine them into a unified, high-performing system.

Methods

UnitedQA uses a three-stage pipeline: Retrieval, Reading, and Re-ranking. Passages are retrieved via Dense Passage Retrieval (DPR) or BM25 + DPR, then evaluated by upgraded reader architectures:

Question ──► Retriever (DPR / BM25) ──► Top 100 Passages
                                              │
             ┌────────────────────────────────┴──────────────────────────────┐
             ▼                                                               ▼
   UnitedQA-E (Extractive)                                         UnitedQA-G (Generative)
   • Backbone: ELECTRA                                            • Backbone: T5 (Fusion-in-Decoder)
   • Multi-passage HardEM + passage-level MML                      • Rank-aware Decoder Attention Bias
   • Posterior Differential Regularization (PDR)                  • Word embedding Adversarial Training
             │                                                               │
             └────────────────────────────────┬──────────────────────────────┘
                                              ▼
                                Re-ranking / Hybrid Voting
                                argmax [ τ·∑ 1(y, y_E) + δ·∑ 1(y, y_G) ]

1. UnitedQA-E (Extractive Reader)

Built on ELECTRA, this module tackles distant-supervision label noise:

  • Hybrid Objective: Combines a multi-passage HardEM loss (optimizing the most probable correct span across all retrieved passages) with passage-level Marginal Log-Likelihood (MML).
  • Posterior Differential Regularization (PDR): Injects Gaussian noise into token embeddings and applies squared Hellinger distance regularization to start/end probability distributions, enhancing robustness against noisy spans.

2. UnitedQA-G (Generative Reader)

Built on T5 using the Fusion-in-Decoder (FiD) architecture:

  • Decoder Attention Bias: Adds a trainable bias vector ($b_k$) to the cross-attention layers, allowing the decoder to explicitly register the retrieval rank order of candidate passages.
  • Adversarial Training: Applies gradient-based perturbations to the word embedding space to improve out-of-domain generalization.

3. Hybrid Combination

During inference, candidate strings from $M$ extractive models and $N$ generative models are merged via weighted vote interpolation: $$\operatorname*{argmax}_{y \in \mathcal{Y}} \ \tau \sum_{m=1}^M \mathbf{1}(y, y^E_m) + \delta \sum_{n=1}^N \mathbf{1}(y, y^G_n)$$ Tuning on the development set yielded optimal weights of $\tau = 0.6, \delta = 0.4$ with an ensemble composed of two generative readers and one extractive reader.

Datasets

The authors evaluate on standard open-domain QA benchmarks using Wikipedia (Dec 20, 2018 dump) broken into 100-word passages:

  • NaturalQuestions (NQ): Information-seeking queries from real Google Search users filtered to short answers ($\le 5$ tokens).
  • TriviaQA: Factoid questions scraped from quiz websites (unfiltered set).
  • EfficientQA Dev Set: Used for fine-grained error and categorical analysis.

No new datasets were introduced.

Key Insights and Ideas

  • Low Cross-Paradigm Agreement: Extractive and generative readers with nearly identical Exact Match (EM) accuracy (~52% on NQ) shared exact string answers on fewer than 50% of test questions. Within the same model family, agreement exceeded 70–80%.
  • Specialized Reasoning Strengths:
  • Generative models excelled at temporal reasoning (answering "when" questions) and handled noisy context better as passage counts scaled (from top-20 to top-100).
  • Extractive models demonstrated superior generalization on queries with zero train-test entity overlap ("No Overlap") and performed better on precise entity lookups ("who" questions).
  • Heterogeneous Beats Homogeneous Ensembling: Combining two generative models and one extractive model significantly outperformed homogeneous 3-model ensembles ($3 \times \text{Generative}$ or $3 \times \text{Extractive}$), proving true complementary behavior.

Key Results

Evaluation is measured by Exact Match (EM) percentage:

Model Architecture Type NQ (EM) TriviaQA (EM)
DPR (Karpukhin et al., 2020) Extractive 41.5 57.9
T5-FID$_{\text{large}}$ (Izacard & Grave, 2021) Generative 51.4 67.6
UnitedQA-E$_{\text{large}}$ (Ours) Single Extractive 51.8 68.9
UnitedQA-G$_{\text{large}}$ (Ours) Single Generative 52.3 68.6
UnitedQA-E++ (Ours) Ensemble ($3\times$ Extractive) 52.4 69.6
UnitedQA-G++ (Ours) Ensemble ($3\times$ Generative) 53.3 69.2
UnitedQA (Ours) Hybrid ($2\times\text{G} + 1\times\text{E}$) 54.7 70.5

UnitedQA outperformed prior state-of-the-art baselines by +3.3 EM on NaturalQuestions and +2.7 EM on TriviaQA.

Limitations and Future Work

  • Inference Overhead: Running two large generative readers alongside an extractive reader increases inference computation roughly threefold relative to a single model.
  • Metric Fragility: Manual error analysis revealed that ~30% of consensus "errors" flagged by automated EM were actually valid answers (due to varying granularity, semantic equivalence, or question ambiguity).
  • Future Directions: The authors highlight compressing the multi-reader pipeline via weight sharing, parameter pruning, quantization, or knowledge distillation—training a single student model under the supervision of the hybrid ensemble teacher.

Summary updated

Abstract

To date, most of recent work under the retrieval-reader framework for open-domain QA focuses on either extractive or generative reader exclusively. In this paper, we study a hybrid approach for leveraging the strengths of both models. We apply novel techniques to enhance both extractive and generative readers built upon recent pretrained neural language models, and find that proper training methods can provide large improvement over previous state-of-the-art models. We demonstrate that a simple hybrid approach by combining answers from both readers can efficiently take advantages of extractive and generative answer inference strategies and outperforms single models as well as homogeneous ensembles. Our approach outperforms previous state-of-the-art models by 3.3 and 2.7 points in exact match on NaturalQuestions and TriviaQA respectively.
BibTeX
@inproceedings{cheng2021unitedqa,
  author = {Hao Cheng and Yelong Shen and Xiaodong Liu and Pengcheng He and Weizhu Chen and Jianfeng Gao},
  title = {{UnitedQA: A Hybrid Approach for Open Domain Question Answering}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
  url = {https://openreview.net/forum?id=VMiYjM08Yq},
}

Loading related content...

Figure