KALM: Knowledge-Aware Integration of Local, Document, and Global Contexts for Long Document Understanding

Shangbin Feng, Zhaoxuan Tan, Wenqian Zhang, Zhenyu Lei, Yulia Tsvetkov

Introduction

Understanding long, complex documents remains a significant challenge for modern Large Language Models (LLMs). While these models excel at processing short text snippets, they often struggle to maintain coherence across lengthy reports or legal documents. The Knowledge-Aware Language Model (KALM) framework addresses this by integrating external knowledge at three distinct scales: local sentences, document-wide entity relationships, and global world knowledge. By utilizing a novel "ContextFusion" layer, KALM achieves state-of-the-art results in high-stakes domains like political bias detection and misinformation spotting.

The Challenge of Long-Document Understanding

Standard LLMs are typically constrained by maximum input lengths, leading to a "short-term memory" problem when processing extensive texts. Furthermore, existing attempts to infuse models with external Knowledge Graphs (KGs) often focus on a single scale—either local word meanings, document-level connections, or broad global facts. This "knowledge silo" prevents models from connecting disparate pieces of information across a long document, resulting in reasoning errors and a lack of nuanced understanding.

The KALM Framework

KALM overcomes these limitations by processing documents through three parallel knowledge-aware tracks:

  1. Local Context: The model identifies key entities and injects their textual descriptions from a KG directly into the local paragraph, clarifying specific mentions.
  2. Document-level Context: Through "knowledge coreference," KALM builds a document graph where paragraphs are nodes connected by shared entities. This allows information to flow based on conceptual relationships rather than simple sequential order.
  3. Global Context: For every mentioned entity, the model extracts a neighborhood of related facts from massive external KGs like Wikidata, providing the necessary "world knowledge" for deep reasoning.
Architecture of the KALM framework showing the integration of local, document, and global contexts through the ContextFusion layer.

The core innovation is the ContextFusion Layer. Rather than simply concatenating these representations at the output, KALM employs an interactive layer using attentive pooling. This mechanism allows the local, document, and global perspectives to "cross-pollinate" their insights before the final prediction is made.

Experimental Setup

KALM was evaluated across six datasets in three critical domains:

  • Political Perspective Detection: Identifying ideological lean in news articles (SemEval, Allsides).
  • Misinformation Detection: Distinguishing between satire, fake news, and trusted reporting (LUN, SLN).
  • Roll Call Vote Prediction: Predicting legislative voting patterns based on bill text and social context.

Analysis of Model Performance

A key finding of the research is KALM's superior performance as document length and "knowledge intensity" (the density of unique entities) increase. Error analysis reveals that KALM dominates in the "top-right corner" of performance graphs—representing long, entity-rich documents—where traditional models frequently fail.

Performance comparison across models based on document length (# sentences) and entity density (# mentioned entities).

Ablation studies confirmed that all three contexts are complementary; removing any one track significantly degrades performance. Furthermore, KALM demonstrates remarkable data efficiency. While standard models suffer sharp accuracy drops when trained on only 10% of available data, KALM maintains steady performance, suggesting that structured knowledge acts as an effective shortcut for learning.

Results and Discussion

KALM achieved State-of-the-Art (SOTA) performance across all evaluated datasets. Notable highlights include:

  • SemEval (Political Bias): 91.45% accuracy, surpassing the previous best of 89.90%.
  • SLN (Misinformation): A Micro-F1 score of 94.22, outperforming specialized graph-based models.

The success of the ContextFusion layer was also validated; replacing it with simpler methods like summation or concatenation led to a drop in performance, proving that the interaction between different context scales is vital for accuracy.

Limitations and Future Directions

Despite its success, KALM faces several hurdles. It is dependent on the quality and coverage of existing Knowledge Graphs; missing information on niche or recent events can limit its effectiveness. Additionally, the system relies on entity linking tools like TagMe, where errors can propagate through the pipeline. Future work will focus on improving computational efficiency for processing thousands of long documents and addressing potential biases inherited from source KGs and LLMs.

Summary updated

Abstract

With the advent of pretrained language models (LMs), increasing research efforts have been focusing on infusing commonsense and domain-specific knowledge to prepare LMs for downstream tasks. These works attempt to leverage knowledge graphs, the de facto standard of symbolic knowledge representation, along with pretrained LMs. While existing approaches have leveraged external knowledge, it remains an open question how to jointly incorporate knowledge graphs representing varying contexts, from local (e.g., sentence), to document-level, to global knowledge, to enable knowledge-rich exchange across these contexts. Such rich contextualization can be especially beneficial for long document understanding tasks since standard pretrained LMs are typically bounded by the input sequence length. In light of these challenges, we propose KALM, a Knowledge-Aware Language Model that jointly leverages knowledge in local, document-level, and global contexts for long document understanding. KALM first encodes long documents and knowledge graphs into the three knowledge-aware context representations. It then processes each context with context-specific layers, followed by a context fusion layer that facilitates knowledge exchange to derive an overarching document representation. Extensive experiments demonstrate that KALM achieves state-of-the-art performance on six long document understanding tasks and datasets. Further analyses reveal that the three knowledge-aware contexts are complementary and they all contribute to model performance, while the importance and information exchange patterns of different contexts vary with respect to different tasks and datasets.
BibTeX
@inproceedings{feng2023kalm,
  author = {Shangbin Feng and Zhaoxuan Tan and Wenqian Zhang and Zhenyu Lei and Yulia Tsvetkov},
  title = {{KALM: Knowledge-Aware Integration of Local, Document, and Global Contexts for Long Document Understanding}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  doi = {10.18653/v1/2023.acl-long.118},
}
Figure