Abstract
KALM (Knowledge-Aware Language Model) is a framework designed to enhance long document understanding by integrating local, document-level, and global contexts. By utilizing a novel ContextFusion layer, KALM enables dynamic interaction between these knowledge scales, achieving state-of-the-art performance in tasks such as misinformation detection and political bias analysis, particularly in data-scarce scenarios.
Introduction
Standard Large Language Models (LMs) like BERT or RoBERTa are limited by fixed input sequence lengths, hindering their ability to grasp nuances in long documents where core ideas are distributed across distant paragraphs. While integrating Knowledge Graphs (KGs) has been explored, most methods focus on a single scale—either local descriptions, document-level links, or global external facts. KALM addresses the lack of a unified framework that allows these three scales of knowledge to interact dynamically during the reasoning process.
Methodology
KALM processes documents through three distinct lenses to capture a comprehensive view of the information:
- Local Context: The model identifies entities in a paragraph and appends their textual descriptions from a knowledge base, providing immediate semantic grounding for specific terms.
- Document-level Context: KALM constructs a "document graph" where paragraphs are nodes connected by shared entities (knowledge coreference). This allows information to flow across the document structure based on conceptual links rather than simple sequence.
- Global Context: The framework extracts a neighborhood of related facts from external KGs (such as Wikidata) for every mentioned entity, providing the necessary world knowledge to connect dots not explicitly mentioned in the text.
At the heart of the architecture is the ContextFusion Layer. Instead of merging representations at the final stage, KALM uses an attentive pooling mechanism and a Transformer encoder to facilitate information exchange between these three contexts at every layer of the model.
Experimental Setup
The researchers evaluated KALM on three high-level reasoning tasks across six datasets:
- Political Perspective Detection: Identifying the lean of news articles using the SemEval and Allsides datasets.
- Misinformation Detection: Spotting "fake news" using the SLN (Satire, Hoax, Propaganda) and LUN (fake vs. real news) datasets.
- Roll Call Vote Prediction: Predicting legislative votes based on bill texts.
Key Results and Insights
KALM outperformed all baselines, including specialized task-specific models and general-purpose long-document models like LongFormer. Several key insights emerged from the evaluation:
- Complementary Knowledge: Ablation studies confirmed that local, document, and global contexts capture non-redundant information; removing any one of them consistently degraded performance.
- Data Efficiency: KALM demonstrated remarkable data efficiency, achieving steady performance with as little as 10% of the training data, whereas standard baselines saw accuracy plummet in low-data regimes.
- Long Document Mastery: As documents become longer and more knowledge-intensive (containing more entities), the performance gap between KALM and standard models grows significantly. This is particularly evident in misinformation detection tasks where external facts are critical.
Conclusion
KALM sets a new state-of-the-art for long document understanding by unifying multi-scale knowledge. While the model currently faces bottlenecks related to entity linking accuracy and knowledge graph sparsity, it provides a robust framework for complex reasoning. Future work aims to explore the integration of other knowledge forms, such as visual or temporal data, into this multi-scale framework.
Summary updated