Encoding Context into LLM Weights: What Makes It Context?

Yue Wang, Qizhou Wang, Feng Liu, Gang Niu, Masashi Sugiyama, Bo HanarXiv 2026 · arXiv

TL;DR

Large Language Models (LLMs) struggle with the massive memory costs of long-context windows. This paper introduces Addressable Context Encoding (ACE), a method that stores context directly into an LLM's weights via test-time updates, using unique identifiers and specialized prompting to ensure the model actually uses that information as active context rather than just "memorizing" it.

What Problem Does This Paper Address?

As we push LLMs to process entire books or massive codebases, we hit a wall: the "KV cache" (the model's short-term memory) grows linearly with context length, while the computation cost grows quadratically.

A promising alternative is Test-time Context Encoding (TCE), where we temporarily update the model's weights to "absorb" the context. However, previous TCE methods had a major flaw: the model often couldn't distinguish between its pre-existing general knowledge and the newly encoded context. It might "memorize" a fact (be able to recite it) but fail to "use" it (apply it to answer a question). Most existing solutions rely on expensive "meta-training" to teach the model how to do this, making them difficult to use with standard, off-the-shelf models.

Methods

The authors propose Addressable Context Encoding (ACE), which works with off-the-shelf LLMs without extra training. It relies on two key components:

  1. Addressable Mechanism (UIDs): During the encoding phase, each chunk of context is paired with a Unique Identifier (UID)—a random string like ctx-573277c6. The model is updated to associate the context with this specific "address."
  2. Prompting Framework:
    • Role-Binding Prompt: During encoding, the model is told: "This text is a context chunk... identified by [UID]. It should be treated as conversational context."
    • Context-Invocation Prompt: During inference, the model is prompted to "Recall context [UID] to continue." This explicitly tells the LLM to treat the information stored at that "address" as the active context for the current query.
Figure 1: Comparison of (a) full-context inference, (b) encoded-context inference with meta-trained TCE, and (c) encoded-context inference with ACE. Full-context inference jointly processes c1:T and x, becoming infeasible when their combined length exceeds the input context window. TCE and ACE instead encode the chunks c1:T into LLM weights, illustrated here as FFN weights. Prior TCE methods often rely on additional meta-training-based fine-tuning, whereas ACE operates directly on off-the-shelf LLMs without such fine-tuning.

By updating only specific "fast weights" (usually the final feed-forward network layer), the authors can inject new information without destroying the model's original capabilities.

Datasets

The researchers evaluated ACE using a mix of controlled "sandbox" tasks and established long-context benchmarks:

  • Sandbox Tasks: Newly curated tests for Recall (retrieving a specific value), Recognition (identifying what was just told), Continuation (answering a pending question), and Conflict (handling information that contradicts the model's internal training data).
  • RULER: A "Needle-in-a-Haystack" benchmark that tests retrieval from 1K to 64K tokens.
  • LongBench & LongBench v2: Comprehensive benchmarks for long-document understanding and reasoning.
  • Retention (MMLU): Used to ensure that encoding new context doesn't "break" the model's general knowledge.

Key Insights and Ideas

  • Recitation $\neq$ Contextual Use: A model can be "fine-tuned" on a document until it can recite it word-for-word, yet still fail to use that document to answer a question. ACE bridges this gap by providing an explicit "access path" via UIDs.
  • The "Address" is the Key: Using a UID acts like a database key. It provides a strong signal that helps the model's attention mechanism focus on the specific weight updates associated with that context.
  • Off-the-Shelf Viability: Perhaps the most important finding is that you don't need complex meta-learning to make TCE work. With the right prompting and addressing strategy, standard models like Qwen3.5 or Llama-based models can effectively "learn" context at test-time.

Key Results

ACE showed dramatic improvements over standard TCE across almost every metric:

  • Sandbox Performance: On the Qwen3.5-27B model, ACE increased Recall accuracy from a dismal 1.5% to 75.6% and Recognition from 0.0% to 98.0%.
  • Handling Conflicts: ACE was remarkably good at prioritizing encoded context over pre-existing knowledge. In "Counterfactual" tasks (where the context lies to the model), ACE achieved over 90% accuracy in following the context, whereas standard TCE often defaulted back to its original training.
Table 1: Comparison between TCE and ACE on Recall, Recognition, Continuation, and Retention. We reported accuracy (%) on the Qwen3.5 family. Bold marks the best result within each model and task.
  • Long-Context Benchmarks: On the RULER NIAH-3 task, ACE improved accuracy from 8% to 63% using the Muse-Glimmer-30B model.
Table 3: Comparison between TCE and ACE on RULER, LONGBENCH, and LONGBENCH V2. We reported retrieval accuracy (%) on RULER, answer F1 (×100) on LONGBENCH, and summed judge-assigned scores over 100 instances on LONGBENCH V2. By default, we use ci_v2. Gold Standard refers to full-context inference with the context retained in the input. Bold marks the better result between TCE and ACE.

Limitations and Future Work

  • Computational Overhead: While ACE saves memory (KV cache), it requires backpropagation during the encoding phase. This is currently slower than simply filling a KV cache for shorter contexts.
  • Reasoning Gap: While ACE is excellent at retrieval, it still lags behind "Gold Standard" (full-context) inference on complex reasoning tasks like LongBench v2.
  • Scaling Updates: The authors noted that while updating more layers can help, it's a balancing act; too many updates can start to degrade the model's general reasoning (the "Retention" problem).

Future research will likely focus on making the weight updates more efficient and improving the model's ability to perform multi-step reasoning over encoded information.

Summary updated

Abstract

Longer contexts allow large language models (LLMs) to condition their predictions on more information, but inference costs grow prohibitively with context length. An emerging alternative uses test-time updates to encode context as temporary changes to model weights. However, encoding this information into weights that already represent general knowledge obscures its role as context, so the LLM may fail to use it properly to condition subsequent inference. To restore this contextual role, we first associate each context with an identifier during encoding, thereby distinguishing the newly encoded information from pre-existing knowledge. Invoking the identifier at inference then enables the LLM to recite the encoded context. Nevertheless, successful recitation does not ensure that the LLM will use the recalled information as context. We therefore further introduce a prompting framework that explicitly guides the LLM to use this information as context. Across diverse controlled tasks and practical benchmarks, our method outperformed the baseline using only off-the-shelf LLMs, showing its advances in context encoding.
BibTeX
@misc{wang2026encoding,
  author = {Yue Wang and Qizhou Wang and Feng Liu and Gang Niu and Masashi Sugiyama and Bo Han},
  title = {{Encoding Context into LLM Weights: What Makes It Context?}},
  year = {2026},
  eprint = {2610.17686},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.17686},
}
Figure