Abstract
Large Language Models (LLMs) frequently struggle with knowledge cutoffs and hallucinations regarding niche or real-time information. While Retrieval-Augmented Generation (RAG) is a common remedy, it faces limitations in context window size and computational efficiency. This paper introduces SR-KI (Scalable and Real-Time Knowledge Integration), a method that injects external knowledge directly into an LLM's internal latent space. By employing "supervised attention," SR-KI enables models to access up to 40,000 facts in real-time with 99.75% data compression and 98% retrieval accuracy on a single GPU.
Introduction
Current approaches to updating LLM knowledge bases—such as fine-tuning, RAG, and long-context modeling—present significant trade-offs. Fine-tuning is computationally expensive and prone to catastrophic forgetting. RAG depends on external search latency and expensive prompt token usage. Long-context models suffer from quadratic scaling costs. SR-KI addresses these challenges by treating the model's attention mechanism as a high-precision internal retriever, making knowledge injection scalable, precise, and interpretable.
Methodology
SR-KI represents a Knowledge Base (KB) as a collection of (subject, relation, object) triples. The integration process involves several key innovations:
- KV Injection: Triples are converted into Key-Value pairs where the
(subject, relation)serves as the Key and the(object)as the Value. These are embedded and projected into the LLM’s internal dimensions. - Retrieval Layer Identification: Empirical analysis reveals a "sweet spot" within LLM architectures—a specific layer where knowledge injection is most effective (e.g., layer 25 in a 32-layer model). This is designated as the Retrieval Layer.
- Supervised Attention Training: To prevent the model from becoming "confused" by a large volume of facts, a specialized loss function is used. This "supervision" explicitly trains the attention mechanism to focus on relevant KB entries for a given query.
- Two-Stage Inference: At the Retrieval Layer, the model calculates attention scores for the entire KB but retains only the top-100 most relevant facts. These selected facts are then reused in all subsequent layers, drastically reducing memory and compute overhead.
Experimental Setup
The researchers evaluated SR-KI using a massive benchmark derived from Wikidata, comprising over 140,000 triples and 150,000 QA pairs. The dataset included single-entity facts, multi-entity comparisons, and "unanswerable" questions to test the model's robustness and ability to admit ignorance. A generalization set using Wikidata aliases was also used to ensure conceptual understanding rather than simple string matching.
Results and Discussion
SR-KI demonstrates superior scalability and efficiency compared to previous internal injection methods like KBLaM.
- Scalability and Memory Efficiency: SR-KI successfully integrated 40,000 facts into a 7B model on a single A100 GPU. As shown in the performance metrics, SR-KI maintains a much lower memory footprint than KBLaM or standard in-context learning as the KB size increases.
- Accuracy and Grounding: The model maintained over 98% Recall@10 on single-entity tasks. By injecting "Reference IDs," the model can output specific source markers (e.g., "[A]"), enhancing the interpretability of its generations.
- The Retrieval Layer Phenomenon: The sharp peak in layer-wise scores confirms that specific layers are structurally predisposed to act as retrieval hubs.
Despite these successes, a notable limitation is the "I Don't Know" problem: supervised attention training slightly reduced the model's accuracy on unanswerable questions, making it more prone to attempting an answer rather than refusing.
Conclusion
SR-KI proves that an LLM’s internal attention mechanism can be transformed into a highly efficient, scalable retriever. By achieving massive compression without sacrificing accuracy, it offers a viable path for real-time knowledge integration. Future work will focus on improving multi-hop reasoning and extending the framework to multimodal data.
Summary updated