Abstract
As the textual data given as the context of various tasks lengthens, having necessary information scattered throughout makes it more difficult for large language models (LLMs) to capture relevant details. This challenge is particularly prominent in tasks such as question answering (QA), where key information is often not evenly distributed within the context. This problem of information sparsity has led to the attempts of various approaches, such as direct context adjustment and retrieval-based methods. However, these approaches typically leverage compressed contexts, which increases the risk that key information may be contained in the dropped portions. To address this issue, we propose the Highlighting entity-AWare Knowledge (HAWK) framework. HAWK consists of three main steps: i) entity extraction, ii) entity-aware subcontext selection, and iii) triplet construction. The core mechanism of HAWK is to highlight key information in a context and structuralize it in an entity-aware manner, facilitating knowledge-enhanced generation. Through extensive experiments, HAWK confirms significant improvements in QA tasks with long contexts, achieving up to a 27.6-point F1 score increase and an average win rate of 76.75% over existing methods.
The Challenge of Information Sparsity
As LLM context windows expand to 128K tokens and beyond, a paradox emerges: more data does not always yield better performance. This is primarily due to information sparsity, where critical evidence is buried under mountains of irrelevant text.
Existing solutions generally fall into two categories:
- Direct Adjustment (Pruning/Windowing): Reducing text size, which risks discarding the "needle" in the haystack.
- Context Reconstruction (Retrieval/Summarization): Selecting chunks or summarizing, which can lose nuanced relationships between distant pieces of information.
The authors identify a "lost in the middle" phenomenon where LLMs struggle to attend to salient details when they are scattered. HAWK aims to bridge this gap by highlighting and structuring key information rather than simply deleting the surrounding context.
The HAWK Framework
HAWK operates in four distinct phases to "densify" the context without losing key details:
- Entity Extraction: The system identifies "head entities" directly from the user's query.
- Entity-aware Subcontext Selection: Instead of retrieving chunks based on the whole query, HAWK performs a multi-view search. It identifies specific sentences or chunks relevant to each extracted head entity, ensuring a more diverse set of evidence.
- Triplet Construction:
- Tail Entity Acquisition: Potential "tail entities" are extracted from the selected subcontexts.
- Multi-agent Filtering: To ensure quality, HAWK uses a "committee" of LLMs (e.g., GPT-4, Claude 3.5) to score entity importance. Only those above a threshold are kept, effectively removing distraction.
- Predicate Generation: The model generates relationships (predicates) between head and filtered tail entities to form structured
(subject, relation, object)triplets. - Answer Generation: The final prompt includes the original context (with entities highlighted using special tokens), the newly constructed knowledge triplets, and the query.
Experimental Results
HAWK was evaluated using Natural Questions (NQ) and MuSiQue datasets across various LLM backbones (ChatGPT, Claude 3.5, LLaMA 3.1, and Mistral):
- Performance Leap: LLaMA 3.1-8B saw a 27.6-point F1 score increase over the vanilla baseline.
- Win Rates: In head-to-head comparisons, HAWK achieved an average win rate of 76.75% against existing methods.
- Multi-hop Superiority: On the MuSiQue dataset, HAWK outperformed strong baselines like GraphRAG, achieving an F1 score of 52.36 compared to GraphRAG's 31.48.
- Length Resilience: While other methods degraded as context length increased, HAWK’s performance remained robust across 4k, 8k, and 16k token lengths.
Conclusion
The core insight of HAWK is that LLMs benefit more from better-organized text than simply less text. By providing a structured Knowledge Graph (KG) alongside raw text, HAWK provides a "reasoning map" for the model. While the multi-agent filtering adds some computational overhead, the significant gains in accuracy and the reduction of noise (filtering out up to 52% of candidate entities) demonstrate the framework's effectiveness in managing long-context information sparsity.
Summary updated