KS-LLM: Knowledge Selection of Large Language Models with Evidence Document for Question Answering

Xinxin Zheng, Feihu Che, Jinyang Wu, Shuai Zhang, Shuai Nie, Kang Liu, Jianhua TaoarXiv 2024 · DOI · arXiv

Introduction

Large Language Models (LLMs) have demonstrated remarkable capabilities in natural language understanding and generation. However, they frequently struggle with "hallucinations"—generating factually incorrect information—especially in knowledge-intensive tasks like open-domain question answering (QA). A common strategy to mitigate this is Retrieval-Augmented Generation (RAG), where the model is provided with an external evidence document to ground its response.

While providing context helps, it introduces a new challenge: information overload. Evidence documents, whether retrieved from Wikipedia or generated by other models, often contain significant noise and irrelevant details. When LLMs are forced to process long, noisy documents, their performance often degrades due to the "lost in the middle" phenomenon or simple confusion. This paper introduces KS-LLM (Knowledge Selection of Large Language Models), a framework designed to intelligently filter evidence documents into concise, highly relevant snippets before they reach the generation stage.

Methodology: The KS-LLM Pipeline

The core of KS-LLM is a three-stage pipeline that transforms a raw evidence document into a refined set of knowledge inputs.

The KS-LLM pipeline: Triple Construction, Evidence Sentence Selection, and Answer Generation.

1. Triple Construction

Rather than using the user's raw question as a search query, KS-LLM first prompts an LLM to extract the subject entity and generate structured knowledge graph triples. For example, the question "What star sign is Jamie Lee Curtis?" might yield triples like (Jamie Lee Curtis, occupation, actress) and (Jamie Lee Curtis, birthdate, November 22, 1958). This step expands the semantic representation of the question, making it easier to match against relevant parts of the document.

2. Evidence Sentence Selection

The system then uses a BERT-based encoder to create dense vector embeddings for both the generated triples and every individual sentence in the evidence document. By calculating the Euclidean distance between these embeddings, the system identifies the top-$k$ sentences that are most semantically similar to the triples. This effectively filters out the "noise" of the full document.

3. Answer Generation

In the final stage, the LLM receives the original question, the structured triples, and the selected evidence sentences. By combining structured relational data with unstructured contextual sentences, the model can reason more accurately and efficiently.

Experimental Setup and Datasets

The authors evaluated KS-LLM across three prominent open-domain QA benchmarks:

  • TriviaQA-verified: A dataset featuring manually verified question-answer-evidence sets.
  • WebQ (WebQuestions): Real-world questions sourced from the web.
  • NQ (Natural Questions): Queries derived from Google search logs.

For datasets lacking default evidence documents (WebQ and NQ), the authors utilized Vicuna-13B to generate synthetic evidence documents, testing the model's ability to extract truth even from potentially hallucinated sources.

Key Insights and Analysis

The researchers' analysis revealed several critical findings regarding how LLMs handle context:

  • The Cost of Noise: An ablation study on document length showed that performance drops significantly as more context is added. Accuracy fell from ~53% with a 300-token document to ~37% with a 2,000-token document, highlighting the necessity of filtering.
  • Structured vs. Unstructured Synergy: The model achieves its highest accuracy when it has access to both triples and sentences. Triples provide a clear logical anchor, while sentences provide the necessary linguistic context.
  • The "Goldilocks" Zone for Evidence: Testing different values for $k$ (the number of selected sentences) showed that $k=2$ is optimal. Fewer sentences provide insufficient context, while more sentences re-introduce distracting noise.

Results

KS-LLM was tested using Vicuna-13B, Llama 2-13B, and Llama 2-7B. Across all configurations, the method consistently outperformed standard baselines.

Performance comparison (Exact Match) of KS-LLM against various baselines across three datasets.

As shown in the table above, KS-LLM achieved a peak Exact Match (EM) score of 58.48 on TriviaQA-verified using Vicuna-13B, a significant improvement over the standard document-augmented baseline (52.69). Similar gains were observed in WebQ and NQ, where KS-LLM (using Llama 2-13B) reached EM scores of 24.70 and 21.69, respectively.

Beyond accuracy, KS-LLM offers substantial efficiency gains. By reducing the context window from a full document to just a few sentences, inference time was reduced by more than 50%.

Conclusion and Future Work

KS-LLM demonstrates that for LLMs, "less is more" when it comes to evidence documents. By intelligently selecting only the most relevant sentences guided by structured triples, the framework mitigates hallucination and improves computational efficiency.

Limitations: The reliance on an external embedding model (BERT) adds architectural complexity. Furthermore, when using LLM-generated evidence (as in WebQ/NQ), the system is still bounded by the quality of the initial generation.

Future Directions: Future research could explore dynamic selection mechanisms where the model determines the optimal number of evidence sentences based on the complexity of the query. Additionally, applying KS-LLM to multi-hop reasoning tasks could further validate its ability to handle complex information retrieval.

Summary updated

Abstract

Large language models (LLMs) suffer from the hallucination problem and face significant challenges when applied to knowledge-intensive tasks. A promising approach is to leverage evidence documents as extra supporting knowledge, which can be obtained through retrieval or generation. However, existing methods directly leverage the entire contents of the evidence document, which may introduce noise information and impair the performance of large language models. To tackle this problem, we propose a novel Knowledge Selection of Large Language Models (KS-LLM) method, aiming to identify valuable information from evidence documents. The KS-LLM approach utilizes triples to effectively select knowledge snippets from evidence documents that are beneficial to answering questions. Specifically, we first generate triples based on the input question, then select the evidence sentences most similar to triples from the evidence document, and finally combine the evidence sentences and triples to assist large language models in generating answers. Experimental comparisons on several question answering datasets, such as TriviaQA, WebQ, and NQ, demonstrate that the proposed method surpasses the baselines and achieves the best results.
BibTeX
@misc{zheng2024ks,
  author = {Xinxin Zheng and Feihu Che and Jinyang Wu and Shuai Zhang and Shuai Nie and Kang Liu and Jianhua Tao},
  title = {{KS-LLM: Knowledge Selection of Large Language Models with Evidence Document for Question Answering}},
  year = {2024},
  eprint = {2404.15660},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2404.15660},
}
Figure