Abstract
To date, most of recent work under the retrieval-reader framework for open-domain QA focuses on either extractive or generative reader exclusively. In this paper, we study a hybrid approach for leveraging the strengths of both models. We apply novel techniques to enhance both extractive and generative readers built upon recent pretrained neural language models, and find that proper training methods can provide large improvement over previous state-of-the-art models. We demonstrate that a simple hybrid approach by combining answers from both readers can efficiently take advantages of extractive and generative answer inference strategies and outperforms single models as well as homogeneous ensembles. Our approach outperforms previous state-of-the-art models by 3.3 and 2.7 points in exact match on NaturalQuestions and TriviaQA respectively.
Introduction
In Open-Domain Question Answering (ODQA), systems typically follow a "retriever-reader" pipeline: a retriever identifies relevant snippets from a massive corpus like Wikipedia, and a "reader" model inspects those snippets to extract or generate the answer. Historically, research has diverged into two paradigms: extractive readers, which identify exact text spans, and generative readers, which synthesize answers token-by-token. While extractive models are grounded in text, generative models offer flexibility but risk hallucinations. UnitedQA bridges this gap by ensembling improved versions of both reader types, leveraging their complementary strengths.
Methods
UnitedQA-E (The Extractive Reader)
Built on the ELECTRA language model, UnitedQA-E introduces two key improvements:
- Multi-objective Training: By combining "HardEM" (learning from the most likely correct span) with Marginal Log-Likelihood (MML), the model better handles cases where the answer appears multiple times in the text, distinguishing relevant instances from coincidental matches.
- Posterior Differential Regularization (PDR): To improve robustness against noisy labels, the authors injected small amounts of noise into the input during training and forced the model to remain consistent in its predictions.
UnitedQA-G (The Generative Reader)
Based on the T5 "Fusion-in-Decoder" (FiD) architecture, UnitedQA-G incorporates:
- Decoder Attention Bias: This mechanism informs the decoder of the retriever's passage rankings, helping the model prioritize high-relevance evidence over distractor passages.
- Adversarial Training: Small perturbations added to word embeddings during training enhance the model's ability to generalize to unseen data.
The Hybrid System
The final UnitedQA system selects answers via a simple linear interpolation (a weighted vote) between the top predictions of both the extractive and generative components.
Key Insights and Ideas
The "Agreement" Gap
A striking finding of this research is that extractive and generative models agree on the answer less than 50% of the time. This low agreement indicates that the models are not redundant but highly complementary; when one paradigm fails, the other often succeeds.
Model Specialization
The authors observed distinct specializations: generative models excel at temporal reasoning (dates and sequences), while extractive models demonstrate superior generalization on topics less frequent in the training data. Furthermore, the inclusion of Attention Bias in the generative model proved that explicitly ranking retrieved passages significantly boosts performance by filtering noise.
Key Results
UnitedQA established new state-of-the-art (SOTA) benchmarks on two gold-standard ODQA datasets: NaturalQuestions (NQ) and TriviaQA.
As shown in the results table, UnitedQA achieved an Exact Match (EM) score of 54.7 on NQ and 70.5 on TriviaQA. This outperforms the previous best (T5-FiD) by 3.3 and 2.7 points, respectively. Notably, the hybrid model also outperformed "homogeneous ensembles" (e.g., ensembling three generative models), proving that architectural diversity is more valuable than simply increasing the count of a single model type.
Limitations and Future Work
While highly effective, the hybrid approach introduces an "inference tax," requiring roughly three times the computational power of a single generative model to run both large-scale readers. Future research directions include distilling this hybrid knowledge into a single, smaller model for real-time applications and addressing the retriever bottleneck, as the system remains dependent on the quality of the initial document search.
Summary updated