SR-KI: Scalable and Real-Time Knowledge Integration into LLMs via Supervised Attention

Kang Liu (刘康), Bohan Yu, Wei Huang, Kang LiuarXiv 2025 · DOI · arXiv

Abstract

Large Language Models (LLMs) frequently struggle with knowledge cutoffs and hallucinations regarding niche or real-time information. While Retrieval-Augmented Generation (RAG) is a common remedy, it faces limitations in context window size and computational efficiency. This paper introduces SR-KI (Scalable and Real-Time Knowledge Integration), a method that injects external knowledge directly into an LLM's internal latent space. By employing "supervised attention," SR-KI enables models to access up to 40,000 facts in real-time with 99.75% data compression and 98% retrieval accuracy on a single GPU.

Introduction

Current approaches to updating LLM knowledge bases—such as fine-tuning, RAG, and long-context modeling—present significant trade-offs. Fine-tuning is computationally expensive and prone to catastrophic forgetting. RAG depends on external search latency and expensive prompt token usage. Long-context models suffer from quadratic scaling costs. SR-KI addresses these challenges by treating the model's attention mechanism as a high-precision internal retriever, making knowledge injection scalable, precise, and interpretable.

Methodology

SR-KI represents a Knowledge Base (KB) as a collection of (subject, relation, object) triples. The integration process involves several key innovations:

  1. KV Injection: Triples are converted into Key-Value pairs where the (subject, relation) serves as the Key and the (object) as the Value. These are embedded and projected into the LLM’s internal dimensions.
  2. Retrieval Layer Identification: Empirical analysis reveals a "sweet spot" within LLM architectures—a specific layer where knowledge injection is most effective (e.g., layer 25 in a 32-layer model). This is designated as the Retrieval Layer.
  3. Supervised Attention Training: To prevent the model from becoming "confused" by a large volume of facts, a specialized loss function is used. This "supervision" explicitly trains the attention mechanism to focus on relevant KB entries for a given query.
  4. Two-Stage Inference: At the Retrieval Layer, the model calculates attention scores for the entire KB but retains only the top-100 most relevant facts. These selected facts are then reused in all subsequent layers, drastically reducing memory and compute overhead.
SR-KI Architecture and Supervised Attention Training mechanism.

Experimental Setup

The researchers evaluated SR-KI using a massive benchmark derived from Wikidata, comprising over 140,000 triples and 150,000 QA pairs. The dataset included single-entity facts, multi-entity comparisons, and "unanswerable" questions to test the model's robustness and ability to admit ignorance. A generalization set using Wikidata aliases was also used to ensure conceptual understanding rather than simple string matching.

Results and Discussion

SR-KI demonstrates superior scalability and efficiency compared to previous internal injection methods like KBLaM.

  • Scalability and Memory Efficiency: SR-KI successfully integrated 40,000 facts into a 7B model on a single A100 GPU. As shown in the performance metrics, SR-KI maintains a much lower memory footprint than KBLaM or standard in-context learning as the KB size increases.
  • Accuracy and Grounding: The model maintained over 98% Recall@10 on single-entity tasks. By injecting "Reference IDs," the model can output specific source markers (e.g., "[A]"), enhancing the interpretability of its generations.
  • The Retrieval Layer Phenomenon: The sharp peak in layer-wise scores confirms that specific layers are structurally predisposed to act as retrieval hubs.
Quantitative analysis of layer-wise performance, GPU memory scalability, and unanswerable accuracy.

Despite these successes, a notable limitation is the "I Don't Know" problem: supervised attention training slightly reduced the model's accuracy on unanswerable questions, making it more prone to attempting an answer rather than refusing.

Conclusion

SR-KI proves that an LLM’s internal attention mechanism can be transformed into a highly efficient, scalable retriever. By achieving massive compression without sacrificing accuracy, it offers a viable path for real-time knowledge integration. Future work will focus on improving multi-hop reasoning and extending the framework to multimodal data.

Summary updated

BibTeX
@misc{2025sr,
  author = {Kang Liu (刘康) and Bohan Yu and Wei Huang and Kang Liu},
  title = {{SR-KI: Scalable and Real-Time Knowledge Integration into LLMs via Supervised Attention}},
  year = {2025},
  eprint = {2511.06446},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2511.06446},
}
Figure