HAWK: Highlighting Entity-aware Knowledge for Alleviating Information Sparsity in Long Contexts

Seonmin Koo, Jinsung Kim, Chanjun Park, Heuiseok Lim2025 · DOI

Abstract

As the textual data given as the context of various tasks lengthens, having necessary information scattered throughout makes it more difficult for large language models (LLMs) to capture relevant details. This challenge is particularly prominent in tasks such as question answering (QA), where key information is often not evenly distributed within the context. This problem of information sparsity has led to the attempts of various approaches, such as direct context adjustment and retrieval-based methods. However, these approaches typically leverage compressed contexts, which increases the risk that key information may be contained in the dropped portions. To address this issue, we propose the Highlighting entity-AWare Knowledge (HAWK) framework. HAWK consists of three main steps: i) entity extraction, ii) entity-aware subcontext selection, and iii) triplet construction. The core mechanism of HAWK is to highlight key information in a context and structuralize it in an entity-aware manner, facilitating knowledge-enhanced generation. Through extensive experiments, HAWK confirms significant improvements in QA tasks with long contexts, achieving up to a 27.6-point F1 score increase and an average win rate of 76.75% over existing methods.

The Challenge of Information Sparsity

As LLM context windows expand to 128K tokens and beyond, a paradox emerges: more data does not always yield better performance. This is primarily due to information sparsity, where critical evidence is buried under mountains of irrelevant text.

Information sparsity and the limitations of existing context adjustment methods.

Existing solutions generally fall into two categories:

  1. Direct Adjustment (Pruning/Windowing): Reducing text size, which risks discarding the "needle" in the haystack.
  2. Context Reconstruction (Retrieval/Summarization): Selecting chunks or summarizing, which can lose nuanced relationships between distant pieces of information.

The authors identify a "lost in the middle" phenomenon where LLMs struggle to attend to salient details when they are scattered. HAWK aims to bridge this gap by highlighting and structuring key information rather than simply deleting the surrounding context.

The HAWK Framework

HAWK operates in four distinct phases to "densify" the context without losing key details:

The HAWK framework architecture, illustrating the pipeline from entity extraction to answer generation.
  1. Entity Extraction: The system identifies "head entities" directly from the user's query.
  2. Entity-aware Subcontext Selection: Instead of retrieving chunks based on the whole query, HAWK performs a multi-view search. It identifies specific sentences or chunks relevant to each extracted head entity, ensuring a more diverse set of evidence.
  3. Triplet Construction:
    • Tail Entity Acquisition: Potential "tail entities" are extracted from the selected subcontexts.
  4. Multi-agent Filtering: To ensure quality, HAWK uses a "committee" of LLMs (e.g., GPT-4, Claude 3.5) to score entity importance. Only those above a threshold are kept, effectively removing distraction.
  5. Predicate Generation: The model generates relationships (predicates) between head and filtered tail entities to form structured (subject, relation, object) triplets.
  6. Answer Generation: The final prompt includes the original context (with entities highlighted using special tokens), the newly constructed knowledge triplets, and the query.

Experimental Results

HAWK was evaluated using Natural Questions (NQ) and MuSiQue datasets across various LLM backbones (ChatGPT, Claude 3.5, LLaMA 3.1, and Mistral):

  • Performance Leap: LLaMA 3.1-8B saw a 27.6-point F1 score increase over the vanilla baseline.
  • Win Rates: In head-to-head comparisons, HAWK achieved an average win rate of 76.75% against existing methods.
  • Multi-hop Superiority: On the MuSiQue dataset, HAWK outperformed strong baselines like GraphRAG, achieving an F1 score of 52.36 compared to GraphRAG's 31.48.
  • Length Resilience: While other methods degraded as context length increased, HAWK’s performance remained robust across 4k, 8k, and 16k token lengths.

Conclusion

The core insight of HAWK is that LLMs benefit more from better-organized text than simply less text. By providing a structured Knowledge Graph (KG) alongside raw text, HAWK provides a "reasoning map" for the model. While the multi-agent filtering adds some computational overhead, the significant gains in accuracy and the reduction of noise (filtering out up to 52% of candidate entities) demonstrate the framework's effectiveness in managing long-context information sparsity.

Summary updated

Abstract

As the textual data given as the context of various tasks lengthens, having necessary information scattered throughout makes it more difficult for large language models (LLMs) to capture relevant details.This challenge is particularly prominent in tasks such as question answering (QA), where key information is often not evenly distributed within the context.This problem of information sparsity has led to the attempts of various approaches, such as direct context adjustment and retrieval-based methods.However, these approaches typically leverage compressed contexts, which increases the risk that key information may be contained in the dropped portions.Therefore, research from the perspective of addressing the information sparsity while not losing key details in contexts is required.To address this issue, we propose Highlighting entity-AWare Knowledge (HAWK) framework.HAWK consists of three main steps: i) entity extraction, ii) entity-aware subcontext selection, and iii) triplet construction.The core mechanism of HAWK is to highlight key information in a context and structuralize it in an entity-aware manner, facilitating knowledgeenhanced generation.Through extensive experiments and comprehensive analysis, HAWK confirms significant improvements in QA tasks with long contexts, achieving up to a 27.6-point F1 score increase and at least an average win rate of 76.75% over existing methods.
BibTeX
@misc{koo2025hawk,
  author = {Seonmin Koo and Jinsung Kim and Chanjun Park and Heuiseok Lim},
  title = {{HAWK: Highlighting Entity-aware Knowledge for Alleviating Information Sparsity in Long Contexts}},
  year = {2025},
  doi = {10.18653/v1/2025.findings-emnlp.708},
}
Figure