Introduction
Understanding long, complex documents remains a significant challenge for modern Large Language Models (LLMs). While these models excel at processing short text snippets, they often struggle to maintain coherence across lengthy reports or legal documents. The Knowledge-Aware Language Model (KALM) framework addresses this by integrating external knowledge at three distinct scales: local sentences, document-wide entity relationships, and global world knowledge. By utilizing a novel "ContextFusion" layer, KALM achieves state-of-the-art results in high-stakes domains like political bias detection and misinformation spotting.
The Challenge of Long-Document Understanding
Standard LLMs are typically constrained by maximum input lengths, leading to a "short-term memory" problem when processing extensive texts. Furthermore, existing attempts to infuse models with external Knowledge Graphs (KGs) often focus on a single scale—either local word meanings, document-level connections, or broad global facts. This "knowledge silo" prevents models from connecting disparate pieces of information across a long document, resulting in reasoning errors and a lack of nuanced understanding.
The KALM Framework
KALM overcomes these limitations by processing documents through three parallel knowledge-aware tracks:
- Local Context: The model identifies key entities and injects their textual descriptions from a KG directly into the local paragraph, clarifying specific mentions.
- Document-level Context: Through "knowledge coreference," KALM builds a document graph where paragraphs are nodes connected by shared entities. This allows information to flow based on conceptual relationships rather than simple sequential order.
- Global Context: For every mentioned entity, the model extracts a neighborhood of related facts from massive external KGs like Wikidata, providing the necessary "world knowledge" for deep reasoning.
The core innovation is the ContextFusion Layer. Rather than simply concatenating these representations at the output, KALM employs an interactive layer using attentive pooling. This mechanism allows the local, document, and global perspectives to "cross-pollinate" their insights before the final prediction is made.
Experimental Setup
KALM was evaluated across six datasets in three critical domains:
- Political Perspective Detection: Identifying ideological lean in news articles (SemEval, Allsides).
- Misinformation Detection: Distinguishing between satire, fake news, and trusted reporting (LUN, SLN).
- Roll Call Vote Prediction: Predicting legislative voting patterns based on bill text and social context.
Analysis of Model Performance
A key finding of the research is KALM's superior performance as document length and "knowledge intensity" (the density of unique entities) increase. Error analysis reveals that KALM dominates in the "top-right corner" of performance graphs—representing long, entity-rich documents—where traditional models frequently fail.
Ablation studies confirmed that all three contexts are complementary; removing any one track significantly degrades performance. Furthermore, KALM demonstrates remarkable data efficiency. While standard models suffer sharp accuracy drops when trained on only 10% of available data, KALM maintains steady performance, suggesting that structured knowledge acts as an effective shortcut for learning.
Results and Discussion
KALM achieved State-of-the-Art (SOTA) performance across all evaluated datasets. Notable highlights include:
- SemEval (Political Bias): 91.45% accuracy, surpassing the previous best of 89.90%.
- SLN (Misinformation): A Micro-F1 score of 94.22, outperforming specialized graph-based models.
The success of the ContextFusion layer was also validated; replacing it with simpler methods like summation or concatenation led to a drop in performance, proving that the interaction between different context scales is vital for accuracy.
Limitations and Future Directions
Despite its success, KALM faces several hurdles. It is dependent on the quality and coverage of existing Knowledge Graphs; missing information on niche or recent events can limit its effectiveness. Additionally, the system relies on entity linking tools like TagMe, where errors can propagate through the pipeline. Future work will focus on improving computational efficiency for processing thousands of long documents and addressing potential biases inherited from source KGs and LLMs.
Summary updated