TL;DR
Most open-domain question answering (QA) systems choose between two reader paradigms: extractive (selecting text spans) or generative (decoding free-form text). UnitedQA demonstrates that these two families make surprisingly different predictions (less than 50% agreement) despite similar accuracies. By introducing tailored training improvements to both readers and combining them with a simple linear re-ranking ensemble, UnitedQA sets a new state-of-the-art on NaturalQuestions and TriviaQA.
What Problem Does This Paper Address?
In retrieval-augmented open-domain QA, an initial retrieval step surfaces candidate passages from a vast corpus (such as Wikipedia), after which a reader module infers the final answer. Research has historically split into two competing reader camps:
- Extractive readers (e.g., DPR, REALM) that identify contiguous token spans within retrieved contexts.
- Generative readers (e.g., T5-FID, RAG) that synthesize answers token by token.
Existing systems commit exclusively to one paradigm. The authors explore whether these architectures develop fundamentally distinct inference strategies, whether their errors are complementary, and how best to combine them into a unified, high-performing system.
Methods
UnitedQA uses a three-stage pipeline: Retrieval, Reading, and Re-ranking. Passages are retrieved via Dense Passage Retrieval (DPR) or BM25 + DPR, then evaluated by upgraded reader architectures:
Question ──► Retriever (DPR / BM25) ──► Top 100 Passages
│
┌────────────────────────────────┴──────────────────────────────┐
▼ ▼
UnitedQA-E (Extractive) UnitedQA-G (Generative)
• Backbone: ELECTRA • Backbone: T5 (Fusion-in-Decoder)
• Multi-passage HardEM + passage-level MML • Rank-aware Decoder Attention Bias
• Posterior Differential Regularization (PDR) • Word embedding Adversarial Training
│ │
└────────────────────────────────┬──────────────────────────────┘
▼
Re-ranking / Hybrid Voting
argmax [ τ·∑ 1(y, y_E) + δ·∑ 1(y, y_G) ]
1. UnitedQA-E (Extractive Reader)
Built on ELECTRA, this module tackles distant-supervision label noise:
- Hybrid Objective: Combines a multi-passage HardEM loss (optimizing the most probable correct span across all retrieved passages) with passage-level Marginal Log-Likelihood (MML).
- Posterior Differential Regularization (PDR): Injects Gaussian noise into token embeddings and applies squared Hellinger distance regularization to start/end probability distributions, enhancing robustness against noisy spans.
2. UnitedQA-G (Generative Reader)
Built on T5 using the Fusion-in-Decoder (FiD) architecture:
- Decoder Attention Bias: Adds a trainable bias vector ($b_k$) to the cross-attention layers, allowing the decoder to explicitly register the retrieval rank order of candidate passages.
- Adversarial Training: Applies gradient-based perturbations to the word embedding space to improve out-of-domain generalization.
3. Hybrid Combination
During inference, candidate strings from $M$ extractive models and $N$ generative models are merged via weighted vote interpolation: $$\operatorname*{argmax}_{y \in \mathcal{Y}} \ \tau \sum_{m=1}^M \mathbf{1}(y, y^E_m) + \delta \sum_{n=1}^N \mathbf{1}(y, y^G_n)$$ Tuning on the development set yielded optimal weights of $\tau = 0.6, \delta = 0.4$ with an ensemble composed of two generative readers and one extractive reader.
Datasets
The authors evaluate on standard open-domain QA benchmarks using Wikipedia (Dec 20, 2018 dump) broken into 100-word passages:
- NaturalQuestions (NQ): Information-seeking queries from real Google Search users filtered to short answers ($\le 5$ tokens).
- TriviaQA: Factoid questions scraped from quiz websites (unfiltered set).
- EfficientQA Dev Set: Used for fine-grained error and categorical analysis.
No new datasets were introduced.
Key Insights and Ideas
- Low Cross-Paradigm Agreement: Extractive and generative readers with nearly identical Exact Match (EM) accuracy (~52% on NQ) shared exact string answers on fewer than 50% of test questions. Within the same model family, agreement exceeded 70–80%.
- Specialized Reasoning Strengths:
- Generative models excelled at temporal reasoning (answering "when" questions) and handled noisy context better as passage counts scaled (from top-20 to top-100).
- Extractive models demonstrated superior generalization on queries with zero train-test entity overlap ("No Overlap") and performed better on precise entity lookups ("who" questions).
- Heterogeneous Beats Homogeneous Ensembling: Combining two generative models and one extractive model significantly outperformed homogeneous 3-model ensembles ($3 \times \text{Generative}$ or $3 \times \text{Extractive}$), proving true complementary behavior.
Key Results
Evaluation is measured by Exact Match (EM) percentage:
| Model | Architecture Type | NQ (EM) | TriviaQA (EM) |
|---|---|---|---|
| DPR (Karpukhin et al., 2020) | Extractive | 41.5 | 57.9 |
| T5-FID$_{\text{large}}$ (Izacard & Grave, 2021) | Generative | 51.4 | 67.6 |
| UnitedQA-E$_{\text{large}}$ (Ours) | Single Extractive | 51.8 | 68.9 |
| UnitedQA-G$_{\text{large}}$ (Ours) | Single Generative | 52.3 | 68.6 |
| UnitedQA-E++ (Ours) | Ensemble ($3\times$ Extractive) | 52.4 | 69.6 |
| UnitedQA-G++ (Ours) | Ensemble ($3\times$ Generative) | 53.3 | 69.2 |
| UnitedQA (Ours) | Hybrid ($2\times\text{G} + 1\times\text{E}$) | 54.7 | 70.5 |
UnitedQA outperformed prior state-of-the-art baselines by +3.3 EM on NaturalQuestions and +2.7 EM on TriviaQA.
Limitations and Future Work
- Inference Overhead: Running two large generative readers alongside an extractive reader increases inference computation roughly threefold relative to a single model.
- Metric Fragility: Manual error analysis revealed that ~30% of consensus "errors" flagged by automated EM were actually valid answers (due to varying granularity, semantic equivalence, or question ambiguity).
- Future Directions: The authors highlight compressing the multi-reader pipeline via weight sharing, parameter pruning, quantization, or knowledge distillation—training a single student model under the supervision of the hybrid ensemble teacher.
Summary updated