Accelerating Real-Time Question Answering via Question Generation

Yuwei Fang, Shuohang Wang, Zhe Gan, Siqi Sun, Jingjing LiuarXiv 2020 · OpenReview

Introduction

State-of-the-art Question Answering (QA) systems, primarily those utilizing Transformer architectures like BERT or RoBERTa, encounter significant latency challenges in production environments. Standard pipelines require on-the-fly encoding of user queries and computationally intensive cross-attention operations against document passages. This creates a bottleneck for web-scale applications demanding millisecond response times. To address this, we present Ocean-Q (an Ocean of Questions), a "Green NLP" framework that eliminates neural network inference from the online path by pre-generating a massive pool of potential question-answer pairs.

Methodology: The Ocean-Q Framework

Ocean-Q shifts the computational burden to an offline "caching" phase, allowing for rapid retrieval during runtime. The process is divided into two distinct stages: offline generation and online matching.

Overview of the Ocean-Q framework, illustrating the offline generation and online matching stages.

Offline QA-pair Generation

The system proactively generates a comprehensive set of questions for a given corpus (e.g., Wikipedia) through three steps:

  1. Candidate Answer Extraction: A RoBERTa-based model identifies potential answer spans within paragraphs.
  2. Question Generation (QG): A UniLM-based model generates diverse questions for each answer.
  3. Verification: A BERT-based QA model filters the generated pairs, ensuring the questions accurately lead back to the intended answers.

To enhance the quality and diversity of the "Ocean," the authors employ Diverse Beam Search to avoid repetitive phrasing and Multi-task Learning across various datasets (SQuAD, HotpotQA, NewsQA) to improve generalization.

Online Question Matching

At runtime, Ocean-Q avoids neural encoding entirely. Instead, it employs:

  • Two-step Ranking: Traditional TF-IDF (via DrQA) identifies relevant documents and paragraphs.
  • n-gram Matching: The system calculates similarity scores based on unique tokens between the user's query and the pre-generated questions. The answer associated with the top-ranked generated question is returned.

Experimental Evaluation

The framework was evaluated on standard benchmarks including SQuAD (v1.1 and Open), HotpotQA, and NewsQA.

Performance and Efficiency

Ocean-Q demonstrates a significant leap in efficiency compared to existing methods:

  • Speed: It is 4x faster than DENSPI (the previous state-of-the-art for fast QA) and 125x faster than DrQA.
  • Latency: On SQuAD-Open, Ocean-Q achieved an average latency of 0.28 seconds per query on a standard CPU, whereas DENSPI-Hybrid required 1.24s and DrQA took 35s.
  • Accuracy: Despite the offline nature, Ocean-Q maintains competitive performance. It achieved 32.7 EM (Exact Match) on SQuAD-Open, outperforming the original DrQA (29.8 EM) while suffering only a marginal ~3.5% drop compared to the much slower DENSPI.

Discussion and Conclusion

The success of Ocean-Q hinges on bridging the "Generative Gap"—the difficulty of matching user phrasing to document content—by generating an exhaustive set of diverse phrasings offline. This approach enables deployment on standard CPUs without GPU acceleration, significantly reducing energy costs and infrastructure requirements.

However, the method introduces a trade-off in storage overhead, requiring space for approximately 200,000 QA pairs per 100 paragraphs. Future work will focus on closing the remaining accuracy gap through more sophisticated generation models and addressing the challenges of maintaining the "Ocean" in dynamic, frequently updated corpora.

Summary updated

Abstract

Although deep neural networks have achieved tremendous success for question answering (QA), they are still suffering from heavy computational and energy cost for real product deployment. Further, existing QA systems are bottlenecked by the encoding time of real-time questions with neural networks, thus suffering from detectable latency in deployment for large-volume traffic. To reduce the computational cost and accelerate real-time question answering (RTQA) for practical usage, we propose to remove all the neural networks from online QA systems, and present Ocean-Q (an Ocean of Questions), which introduces a new question generation (QG) model to generate a large pool of QA pairs offline, then in real time matches an input question with the candidate QA pool to predict the answer without question encoding. Ocean-Q can be readily deployed in existing distributed database systems or search engine for large-scale query usage, and much greener with no additional cost for maintaining large neural networks. Experiments on SQuAD(-open) and HotpotQA benchmarks demonstrate that Ocean-Q is able to accelerate the fastest state-of-the-art RTQA system by 4X times, with only a 3+% accuracy drop.
BibTeX
@misc{fang2020accelerating,
  author = {Yuwei Fang and Shuohang Wang and Zhe Gan and Siqi Sun and Jingjing Liu},
  title = {{Accelerating Real-Time Question Answering via Question Generation}},
  year = {2020},
  eprint = {2009.05167},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2009.05167},
}
Figure