Introduction
State-of-the-art Question Answering (QA) systems, primarily those utilizing Transformer architectures like BERT or RoBERTa, encounter significant latency challenges in production environments. Standard pipelines require on-the-fly encoding of user queries and computationally intensive cross-attention operations against document passages. This creates a bottleneck for web-scale applications demanding millisecond response times. To address this, we present Ocean-Q (an Ocean of Questions), a "Green NLP" framework that eliminates neural network inference from the online path by pre-generating a massive pool of potential question-answer pairs.
Methodology: The Ocean-Q Framework
Ocean-Q shifts the computational burden to an offline "caching" phase, allowing for rapid retrieval during runtime. The process is divided into two distinct stages: offline generation and online matching.
Offline QA-pair Generation
The system proactively generates a comprehensive set of questions for a given corpus (e.g., Wikipedia) through three steps:
- Candidate Answer Extraction: A RoBERTa-based model identifies potential answer spans within paragraphs.
- Question Generation (QG): A UniLM-based model generates diverse questions for each answer.
- Verification: A BERT-based QA model filters the generated pairs, ensuring the questions accurately lead back to the intended answers.
To enhance the quality and diversity of the "Ocean," the authors employ Diverse Beam Search to avoid repetitive phrasing and Multi-task Learning across various datasets (SQuAD, HotpotQA, NewsQA) to improve generalization.
Online Question Matching
At runtime, Ocean-Q avoids neural encoding entirely. Instead, it employs:
- Two-step Ranking: Traditional TF-IDF (via DrQA) identifies relevant documents and paragraphs.
- n-gram Matching: The system calculates similarity scores based on unique tokens between the user's query and the pre-generated questions. The answer associated with the top-ranked generated question is returned.
Experimental Evaluation
The framework was evaluated on standard benchmarks including SQuAD (v1.1 and Open), HotpotQA, and NewsQA.
Performance and Efficiency
Ocean-Q demonstrates a significant leap in efficiency compared to existing methods:
- Speed: It is 4x faster than DENSPI (the previous state-of-the-art for fast QA) and 125x faster than DrQA.
- Latency: On SQuAD-Open, Ocean-Q achieved an average latency of 0.28 seconds per query on a standard CPU, whereas DENSPI-Hybrid required 1.24s and DrQA took 35s.
- Accuracy: Despite the offline nature, Ocean-Q maintains competitive performance. It achieved 32.7 EM (Exact Match) on SQuAD-Open, outperforming the original DrQA (29.8 EM) while suffering only a marginal ~3.5% drop compared to the much slower DENSPI.
Discussion and Conclusion
The success of Ocean-Q hinges on bridging the "Generative Gap"—the difficulty of matching user phrasing to document content—by generating an exhaustive set of diverse phrasings offline. This approach enables deployment on standard CPUs without GPU acceleration, significantly reducing energy costs and infrastructure requirements.
However, the method introduces a trade-off in storage overhead, requiring space for approximately 200,000 QA pairs per 100 paragraphs. Future work will focus on closing the remaining accuracy gap through more sophisticated generation models and addressing the challenges of maintaining the "Ocean" in dynamic, frequently updated corpora.
Summary updated