Efficient Answer Retrieval System (EARS): Combining Local DB Search and Web Search for Generative QA

Nikita Krayko, Ivan Sidorov, Fedor Laputin, Daria Galimzianova, Vasily KonovalovEMNLP 2024

TL;DR

This paper introduces EARS, a production-ready factual question-answering system that effectively combines local knowledge base search with generative AI powered by LLMs and web search. EARS significantly improves answer retrieval by introducing a novel ranking component and a hybrid search approach, leading to a substantial boost in answer usefulness, especially for voice-based interactions.

What Problem Does This Paper Address?

Developing effective virtual assistants hinges on robust factual question-answering (QA) capabilities. These systems need to handle a wide range of queries, from domain-specific inquiries about a company's products and services to general open-domain questions. Existing approaches often struggle to balance the speed and precision of local knowledge base (KB) lookups with the breadth and generative power of large language models (LLMs) and web search. This can lead to either slow, incomplete answers or inaccurate, hallucinated responses. The challenge is to create a system that is both efficient and accurate, providing users with reliable and relevant answers across diverse domains and query types.

Methods

EARS employs a multi-stage pipeline to address factual QA:

  • Preprocessing: User queries undergo preprocessing, including a "Truecase" component to correct capitalization errors, which is crucial as the E5 embedding model used is case-sensitive.
  • Local Knowledge Base Search (Retriever): The system first attempts to retrieve answers from a local knowledge base. This involves encoding queries and KB entries (query-title-passage triplets) using the multilingual E5 embedding model and performing an Approximate Nearest Neighbor (ANN) search using Faiss. The top-10 semantically similar candidates are retrieved.
  • Ranker: If the Retriever finds potential answers, a custom Gradient Boosting model (Ranker) is employed. This Ranker goes beyond simple similarity by incorporating various features, including Retriever similarity, embeddings (processed via LDA and PCA), Natural Language Inference (NLI) logits, Named Entity Recognition (NER) features, and BM25 scores. This component is key to enhancing retrieval quality.
  • Generative Search (LLM): If the local KB search yields insufficient results (i.e., the Ranker's confidence is below a threshold), the system resorts to Generative Search. This involves querying a web search API for relevant snippets, which are then concatenated to form a context. This context, along with the original query, is fed to an LLM (Mistral 7B in this case) to generate an answer. The LLM is specifically prompted to output "No information" if it cannot find a relevant answer in the provided context.

Datasets

  • Local Knowledge Base: The system's local KB is built from approximately 300,000 popular Wikipedia articles (filtered and cleaned) and data from Wikidata for 36,000 of these articles. Additional corporate sources and manually annotated data are also included, resulting in a KB of around 400,000 entries, updated multiple times a month.
  • Training Data: The Truecase model was trained on the Tatoeba dataset.
  • Evaluation Benchmarks:
    • Golden Set: A benchmark set of 1,600 factual questions with human-crafted answers was created for system validation. This set is designed to be representative of user queries and is used for both manual and automatic evaluation.
  • LLM Evaluation: Several LLMs (Mistral 7B, GPT-3.5, GPT-40) were evaluated on the golden set, with results reported for top-5 and top-10 context snippets from web search.
  • Ranker Evaluation: The Ranker's performance was compared against other ranking techniques, including the Retriever alone and a proprietary Cohere cross-encoder.

Key Insights and Ideas

  • Hybrid Approach is Crucial: Combining local KB search with LLM-powered generative search is essential for a comprehensive QA system. The local KB provides speed and precision for known facts, while generative search handles novel or less common queries.
  • The Ranker is a Game-Changer: The custom Gradient Boosting Ranker significantly improves retrieval quality by leveraging a rich set of features beyond simple semantic similarity. This component is critical for selecting the best answer from multiple candidates.
  • LLM Fine-tuning for Context is Key: While the system uses LLMs for generation, the paper notes that fine-tuning them specifically for context following could further enhance performance, especially in reducing "No information" responses.
  • "No Information" as a Signal: The system's ability to explicitly state "No information" when an answer cannot be found is a valuable feature, preventing hallucinations and managing user expectations. However, the objective is to minimize this, as it's considered uninformative.
  • Language Agnosticism: The pipeline's design, particularly the use of multilingual embeddings, makes it adaptable to any language, a significant advantage for global applications.

Key Results

  • The Ranker component enhances local knowledge base retrieval quality by 23%.
  • Incorporating the LLM component leads to a substantial 92.8% boost in the usefulness of voice-based responses.
  • The overall Usefulness of the system increased from 26.65% to 51.39% with the optimal combination of Ranker and LLM.
  • The custom features added to the Ranker increased retrieval quality by an additional 5.3% over the base Ranker.
  • The system achieved significant performance improvements in terms of requests per second (RPS) and response time after optimization and migration to Triton Inference Servers.
  • Mistral 7B, an open-source LLM, showed higher usefulness than proprietary models like GPT-3.5 and GPT-40 in the system's setup, although GPT-40 had a higher "No information" rate.

Limitations and Future Work

  • LLM Fine-tuning: The deployed LLM has not been fine-tuned for context following, which is a planned area for future research to improve answer generation and reduce uninformative responses.
  • Context Window Expansion: The system currently uses only the top 5-10 search results. Future work aims to expand this context by utilizing more top hits and potentially integrating results from multiple web search APIs.
  • Human Evaluation Cost: Relying on human evaluation is costly and time-consuming. The paper highlights the importance of automatic metrics to alleviate this burden.
  • Domain Applicability: The system may not be suitable for domains requiring very long answers (e.g., recipes) or constant data updates (e.g., real-time exchange rates), which might necessitate specialized agents or third-party APIs.
  • LLM Quantization: Investigating LLM quantization techniques is suggested for further performance enhancement.

Summary updated

BibTeX
@inproceedings{krayko2024efficient,
  author = {Nikita Krayko and Ivan Sidorov and Fedor Laputin and Daria Galimzianova and Vasily Konovalov},
  title = {{Efficient Answer Retrieval System (EARS): Combining Local DB Search and Web Search for Generative QA}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
}
Figure