Select-TTT: Query-Conditioned Selective Fast Weight Updates for In-Place Test-Time Training

Yue Wu, Yang Zhang, Weinan Gan, Chongming Gao, Xinsen Guo, Xiaoyuan Li, Fuli FengarXiv 2026 · arXiv

TL;DR

Select-TTT introduces a query-conditioned update rule for In-Place Test-Time Training (TTT), allowing Large Language Models (LLMs) to selectively internalize relevant context into "fast weights" while ignoring irrelevant noise. By repurposing the model's own frozen output head as a zero-parameter relevance signal, it significantly boosts long-context question-answering performance with negligible computational overhead.

What Problem Does This Paper Address?

While modern LLMs can technically process massive context windows, they often suffer from "memory dilution." In long-context scenarios—like multi-session dialogues or lengthy document analysis—the actual answer-bearing evidence is usually a tiny fraction of the total input.

Standard Test-Time Training (TTT) attempts to solve this by letting the model adapt its own parameters (fast weights) as it reads the context. However, the recent In-Place TTT (IP-TTT) approach uses a uniform "write schedule": it updates weights with equal strength for every chunk of text. The authors discovered that this indiscriminate writing causes irrelevant information to drown out the useful bits, often leading to performance that is no better than a standard frozen model.

Figure 1: Motivation experiment on LongMemEval-S (Llama- 3.1-8B-Instruct). (a) Accuracy vs. the answer-chunk learning rate. (b) Accuracy vs. the irrelevant-chunk learning rate. The default In-Place TTT configuration uses 0.3 for both learning rates. The dashed line marks the zero-shot Baseline (33.4%).

Methods

The authors propose Select-TTT, which modulates the learning rate of each context chunk based on its relevance to the user's query.

  1. Relevance Scoring in Logit Space: Instead of using external encoders or complex attention mechanisms, Select-TTT uses the model's existing, frozen LM head. It maps a chunk's hidden states to the vocabulary space and measures the probability mass assigned to the tokens in the user's query (query-conditioned cross-entropy). A low cross-entropy indicates the chunk is highly relevant.
  2. Selective Multipliers: These scores are converted into per-chunk learning-rate multipliers using a temperature-scaled softmax. This design ensures three things:
  3. Soft Writes: No chunk is entirely ignored, guarding against estimation errors.
  4. Budget Preservation: The total "amount" of information written remains constant; only the distribution changes.
  5. Stability: A clipping mechanism prevents any single chunk from over-writing the memory too aggressively.
  6. Efficiency: To keep the process fast and memory-efficient, the authors use blockwise log-sum-exp and direct logit computation for query tokens, avoiding the need to materialize massive logit tensors.
Overview of the Select-TTT framework, showing the scoring, multiplier conversion, and selective update steps.

Datasets

The researchers evaluated Select-TTT using two primary backbones, Llama-3.1-8B-Instruct and Qwen3-4B, across several benchmarks:

  • LongMemEval-S: Focuses on interactive memory in multi-session dialogues (~115K tokens).
  • LoCoMo: Tests reasoning over very long conversations spanning up to 35 sessions.
  • LongCite-45k: A large-scale long-context QA corpus used for the "continual training" experiments to show that the model can learn to be even more effective at selective TTT.

Key Insights and Ideas

  • The LLM is its own best judge: One of the most elegant findings is that a pre-trained LLM's output head is already a well-calibrated relevance measure. You don't need new parameters or supervised training to identify what's important; the model's internal representations already "know."
  • Memory is a Zero-Sum Game: By enforcing "budget preservation," the authors show that TTT is more about where you focus your updates than just how much you update. Uniformly updating everything is essentially adding noise to your memory.
  • Top-k Pooling is Essential: The authors found that a chunk's relevance is best determined by its most relevant portions (top-k mean pooling) rather than an average of all tokens in the chunk. This makes the signal robust to the "filler" words that exist even in relevant paragraphs.

Key Results

Select-TTT consistently outperformed both standard frozen baselines and uniform IP-TTT.

  • Zero-Shot Gains: On LongMemEval-S, Select-TTT improved average accuracy by +3.8% for Llama and +3.4% for Qwen over standard IP-TTT.
  • Continual Training: When the model was further trained to use the selective mechanism, it maintained a persistent lead over standard continual pre-training (CPT) and uniform TTT, reaching a peak accuracy of 37.4% on Qwen3-4B.
  • Negligible Overhead: Efficiency analysis showed that the scoring mechanism adds almost no latency to the prefill stage and has a negligible impact on peak GPU memory usage.
Figure 3: Protocol II: LongMemEval-S overall accuracy vs. continual-training steps for No Training, CPT, IP-TTT, and Select-TTT (Qwen3-4B backbone).

Limitations and Future Work

The authors acknowledge that the query-conditioned cross-entropy is a heuristic and might not be perfectly calibrated across all layers. They suggest that a learned relevance function could be even more powerful, though it would add complexity.

Additionally, the current method uses fixed chunk sizes to maintain hardware efficiency through context parallelism. Future work could explore semantically adaptive chunking to better capture natural information boundaries. Finally, while the results are strong on 4B and 8B models, scaling these findings to 70B+ parameter models and multi-modal settings remains an open area for research.

Summary updated

Abstract

Test-time training (TTT) enables a deployed language model to internalize incoming context on-the-fly into a small set of fast-weight parameters, rather than merely treating it as transient context for attention. In-Place TTT recently made this idea practical for large language models by repurposing the down-projection of standard MLP blocks as fast weights. However, we find that its benefit largely disappears on realistic web long-context question answering. We attribute this to In-Place TTT's uniform write schedule: every chunk is written with equal strength, so irrelevant content dilutes query-relevant information in fast-weight memory. Building on this insight, we propose Select-TTT, a model-agnostic, query-conditioned rule for In-Place TTT that scales each chunk's write strength by its relevance to the user query. This relevance signal requires no supervision and no new parameters: it is read directly from the model's frozen output head, as the probability mass that a chunk's hidden states assign to the query tokens. A temperature-scaled softmax turns these scores into learning-rate multipliers, leaving the chunk-wise, context-parallel computation of In-Place TTT intact. Select-TTT consistently improves long-context question answering on two benchmarks and two backbones. These findings highlight the value of query-conditioned fast weight writes for practical test-time training on long-context question answering, which is also potentially promising for long-horizon agent tasks involving multiple user queries over evolving contexts.
BibTeX
@misc{wu2026select,
  author = {Yue Wu and Yang Zhang and Weinan Gan and Chongming Gao and Xinsen Guo and Xiaoyuan Li and Fuli Feng},
  title = {{Select-TTT: Query-Conditioned Selective Fast Weight Updates for In-Place Test-Time Training}},
  year = {2026},
  eprint = {2610.36794},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.36794},
}
Figure