Introduction
Large Language Models (LLMs) have demonstrated remarkable capabilities in natural language understanding and generation. However, they frequently struggle with "hallucinations"—generating factually incorrect information—especially in knowledge-intensive tasks like open-domain question answering (QA). A common strategy to mitigate this is Retrieval-Augmented Generation (RAG), where the model is provided with an external evidence document to ground its response.
While providing context helps, it introduces a new challenge: information overload. Evidence documents, whether retrieved from Wikipedia or generated by other models, often contain significant noise and irrelevant details. When LLMs are forced to process long, noisy documents, their performance often degrades due to the "lost in the middle" phenomenon or simple confusion. This paper introduces KS-LLM (Knowledge Selection of Large Language Models), a framework designed to intelligently filter evidence documents into concise, highly relevant snippets before they reach the generation stage.
Methodology: The KS-LLM Pipeline
The core of KS-LLM is a three-stage pipeline that transforms a raw evidence document into a refined set of knowledge inputs.
1. Triple Construction
Rather than using the user's raw question as a search query, KS-LLM first prompts an LLM to extract the subject entity and generate structured knowledge graph triples. For example, the question "What star sign is Jamie Lee Curtis?" might yield triples like (Jamie Lee Curtis, occupation, actress) and (Jamie Lee Curtis, birthdate, November 22, 1958). This step expands the semantic representation of the question, making it easier to match against relevant parts of the document.
2. Evidence Sentence Selection
The system then uses a BERT-based encoder to create dense vector embeddings for both the generated triples and every individual sentence in the evidence document. By calculating the Euclidean distance between these embeddings, the system identifies the top-$k$ sentences that are most semantically similar to the triples. This effectively filters out the "noise" of the full document.
3. Answer Generation
In the final stage, the LLM receives the original question, the structured triples, and the selected evidence sentences. By combining structured relational data with unstructured contextual sentences, the model can reason more accurately and efficiently.
Experimental Setup and Datasets
The authors evaluated KS-LLM across three prominent open-domain QA benchmarks:
- TriviaQA-verified: A dataset featuring manually verified question-answer-evidence sets.
- WebQ (WebQuestions): Real-world questions sourced from the web.
- NQ (Natural Questions): Queries derived from Google search logs.
For datasets lacking default evidence documents (WebQ and NQ), the authors utilized Vicuna-13B to generate synthetic evidence documents, testing the model's ability to extract truth even from potentially hallucinated sources.
Key Insights and Analysis
The researchers' analysis revealed several critical findings regarding how LLMs handle context:
- The Cost of Noise: An ablation study on document length showed that performance drops significantly as more context is added. Accuracy fell from ~53% with a 300-token document to ~37% with a 2,000-token document, highlighting the necessity of filtering.
- Structured vs. Unstructured Synergy: The model achieves its highest accuracy when it has access to both triples and sentences. Triples provide a clear logical anchor, while sentences provide the necessary linguistic context.
- The "Goldilocks" Zone for Evidence: Testing different values for $k$ (the number of selected sentences) showed that $k=2$ is optimal. Fewer sentences provide insufficient context, while more sentences re-introduce distracting noise.
Results
KS-LLM was tested using Vicuna-13B, Llama 2-13B, and Llama 2-7B. Across all configurations, the method consistently outperformed standard baselines.
As shown in the table above, KS-LLM achieved a peak Exact Match (EM) score of 58.48 on TriviaQA-verified using Vicuna-13B, a significant improvement over the standard document-augmented baseline (52.69). Similar gains were observed in WebQ and NQ, where KS-LLM (using Llama 2-13B) reached EM scores of 24.70 and 21.69, respectively.
Beyond accuracy, KS-LLM offers substantial efficiency gains. By reducing the context window from a full document to just a few sentences, inference time was reduced by more than 50%.
Conclusion and Future Work
KS-LLM demonstrates that for LLMs, "less is more" when it comes to evidence documents. By intelligently selecting only the most relevant sentences guided by structured triples, the framework mitigates hallucination and improves computational efficiency.
Limitations: The reliance on an external embedding model (BERT) adds architectural complexity. Furthermore, when using LLM-generated evidence (as in WebQ/NQ), the system is still bounded by the quality of the initial generation.
Future Directions: Future research could explore dynamic selection mechanisms where the model determines the optimal number of evidence sentences based on the complexity of the query. Additionally, applying KS-LLM to multi-hop reasoning tasks could further validate its ability to handle complex information retrieval.
Summary updated