Designing a Minimal Retrieve-and-Read System for Open-Domain Question Answering

Sohee Yang, Minjoon Seo

Abstract: Sohee Yang, Minjoon Seo. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2021.

Open-domain question answering (QA) systems typically require significant storage for document corpora and embeddings. This paper introduces "Minimal R&R," a series of optimization strategies that shrink the storage footprint of a standard retrieve-and-read system by 160x—from 77.5 GB to under 500 MB—while maintaining competitive accuracy. By combining aggressive passage filtering, a unified single-encoder architecture, and post-training compression, the authors demonstrate that retrieve-and-read systems can be small enough for edge devices while outperforming purely parametric models like T5.

The Challenge: The Storage Tax of Knowledge

Open-domain QA systems generally fall into two categories:

  1. Retrieve-and-Read (R&R): These systems find relevant documents in a massive corpus (like Wikipedia) and "read" them to extract answers. They are interpretable and easy to update but require massive storage for millions of document embeddings and raw text.
  2. Parametric (Closed-book): Models like T5 store all knowledge within their weights. They are compact but act as "black boxes," making it difficult to correct errors or update information without full retraining.

The authors address this "storage tax," aiming to build a system that retains the flexibility of R&R while fitting within the tight constraints (e.g., <500 MB) of mobile devices or standalone containers.

Methodology: The Minimal R&R Framework

The authors propose three categories of orthogonal strategies to minimize the system footprint:

1. Passage Filtering

Instead of indexing all 21 million passages in Wikipedia, the authors train a binary classifier to identify passages likely to contain answers to factoid questions. This allows them to prune the corpus by over 94%, retaining only the top 1.2 million passages.

2. Unified Retriever-Reader Architecture

Standard systems often use three separate encoders. The authors collapse these into a single MobileBERT backbone (4.3x smaller than BERT-base) using several techniques:

  • Dimension Reduction: Reducing embedding sizes from 768 to 128 dimensions.
  • Encoder Sharing: Using the same weights to encode both the question and the passage.
  • Knowledge Distillation: Distilling the "reading" ability of a large, complex model into the lightweight retriever encoder.

3. Post-Training Compression

Additional engineering optimizations further squeeze the data:

  • INT8 Quantization: Shrinking the search index size by 4x.
  • FP16 Weights: Storing model weights in half-precision.
  • Resource Compression: Using bzip2 and lzma for binary and text files.
  • Token ID Storage: Storing the corpus as pre-tokenized IDs rather than raw strings to save space and reduce inference latency.

Results and Analysis

The primary benchmark for this work is the EfficientQA dataset (specifically the "Systems Under 500MB" track), alongside Natural Questions and TriviaQA.

Accuracy vs. System Footprint: Minimal R&R vs. T5 variants.

As illustrated in Figure 1, the Minimal R&R strategies significantly improve the trade-off between accuracy and system size. A key insight was the power of pruning: discarding 94% of the corpus resulted in only a ~2.7% drop in accuracy, proving to be the most effective single reduction method.

Comparison of system size and accuracy across different QA architectures.

The final system achieved a 160x reduction in size compared to the baseline DPR system (77.5 GB reduced to 484.69 MB). As shown in Figure 2, at the 500MB scale, the Minimal R&R system (32% EM) significantly outperformed purely parametric models like T5-small (18% EM). Consequently, the system achieved 1st place in human evaluation and 2nd place in automatic evaluation in the EfficientQA "Under 500MB" competition track.

Conclusion and Future Work

The Minimal R&R approach demonstrates that retrieve-and-read systems can be optimized for edge deployment without sacrificing significant performance. However, challenges remain regarding "stale indices" during training—where pre-computed embeddings may become outdated as model weights evolve. Future work may involve refreshing the index dynamically during the training process to further boost performance.

Summary updated

BibTeX
@inproceedings{yang2021designing,
  author = {Sohee Yang and Minjoon Seo},
  title = {{Designing a Minimal Retrieve-and-Read System for Open-Domain Question Answering}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
  url = {https://openreview.net/forum?id=PJYf3GLNKu},
}
Figure