UnitedQA: A Hybrid Approach for Open Domain Question Answering

Hao Cheng, Yelong Shen, Xiaodong Liu, Pengcheng He, Weizhu Chen, Jianfeng Gao

Abstract

To date, most of recent work under the retrieval-reader framework for open-domain QA focuses on either extractive or generative reader exclusively. In this paper, we study a hybrid approach for leveraging the strengths of both models. We apply novel techniques to enhance both extractive and generative readers built upon recent pretrained neural language models, and find that proper training methods can provide large improvement over previous state-of-the-art models. We demonstrate that a simple hybrid approach by combining answers from both readers can efficiently take advantages of extractive and generative answer inference strategies and outperforms single models as well as homogeneous ensembles. Our approach outperforms previous state-of-the-art models by 3.3 and 2.7 points in exact match on NaturalQuestions and TriviaQA respectively.

Introduction

In Open-Domain Question Answering (ODQA), systems typically follow a "retriever-reader" pipeline: a retriever identifies relevant snippets from a massive corpus like Wikipedia, and a "reader" model inspects those snippets to extract or generate the answer. Historically, research has diverged into two paradigms: extractive readers, which identify exact text spans, and generative readers, which synthesize answers token-by-token. While extractive models are grounded in text, generative models offer flexibility but risk hallucinations. UnitedQA bridges this gap by ensembling improved versions of both reader types, leveraging their complementary strengths.

Methods

UnitedQA-E (The Extractive Reader)

Built on the ELECTRA language model, UnitedQA-E introduces two key improvements:

  • Multi-objective Training: By combining "HardEM" (learning from the most likely correct span) with Marginal Log-Likelihood (MML), the model better handles cases where the answer appears multiple times in the text, distinguishing relevant instances from coincidental matches.
  • Posterior Differential Regularization (PDR): To improve robustness against noisy labels, the authors injected small amounts of noise into the input during training and forced the model to remain consistent in its predictions.

UnitedQA-G (The Generative Reader)

Based on the T5 "Fusion-in-Decoder" (FiD) architecture, UnitedQA-G incorporates:

  • Decoder Attention Bias: This mechanism informs the decoder of the retriever's passage rankings, helping the model prioritize high-relevance evidence over distractor passages.
  • Adversarial Training: Small perturbations added to word embeddings during training enhance the model's ability to generalize to unseen data.

The Hybrid System

The final UnitedQA system selects answers via a simple linear interpolation (a weighted vote) between the top predictions of both the extractive and generative components.

Key Insights and Ideas

The "Agreement" Gap

A striking finding of this research is that extractive and generative models agree on the answer less than 50% of the time. This low agreement indicates that the models are not redundant but highly complementary; when one paradigm fails, the other often succeeds.

Agreement heatmap showing low correlation between extractive (E) and generative (G) readers.

Model Specialization

The authors observed distinct specializations: generative models excel at temporal reasoning (dates and sequences), while extractive models demonstrate superior generalization on topics less frequent in the training data. Furthermore, the inclusion of Attention Bias in the generative model proved that explicitly ranking retrieved passages significantly boosts performance by filtering noise.

Key Results

UnitedQA established new state-of-the-art (SOTA) benchmarks on two gold-standard ODQA datasets: NaturalQuestions (NQ) and TriviaQA.

Comparison of UnitedQA against previous SOTA models on NQ and TriviaQA.

As shown in the results table, UnitedQA achieved an Exact Match (EM) score of 54.7 on NQ and 70.5 on TriviaQA. This outperforms the previous best (T5-FiD) by 3.3 and 2.7 points, respectively. Notably, the hybrid model also outperformed "homogeneous ensembles" (e.g., ensembling three generative models), proving that architectural diversity is more valuable than simply increasing the count of a single model type.

Limitations and Future Work

While highly effective, the hybrid approach introduces an "inference tax," requiring roughly three times the computational power of a single generative model to run both large-scale readers. Future research directions include distilling this hybrid knowledge into a single, smaller model for real-time applications and addressing the retriever bottleneck, as the system remains dependent on the quality of the initial document search.

Summary updated

Abstract

To date, most of recent work under the retrieval-reader framework for open-domain QA focuses on either extractive or generative reader exclusively. In this paper, we study a hybrid approach for leveraging the strengths of both models. We apply novel techniques to enhance both extractive and generative readers built upon recent pretrained neural language models, and find that proper training methods can provide large improvement over previous state-of-the-art models. We demonstrate that a simple hybrid approach by combining answers from both readers can efficiently take advantages of extractive and generative answer inference strategies and outperforms single models as well as homogeneous ensembles. Our approach outperforms previous state-of-the-art models by 3.3 and 2.7 points in exact match on NaturalQuestions and TriviaQA respectively.
BibTeX
@inproceedings{cheng2021unitedqa,
  author = {Hao Cheng and Yelong Shen and Xiaodong Liu and Pengcheng He and Weizhu Chen and Jianfeng Gao},
  title = {{UnitedQA: A Hybrid Approach for Open Domain Question Answering}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
  url = {https://openreview.net/forum?id=VMiYjM08Yq},
}
Figure