Let the Gate Decide: Per-Lookup Precision for Quantized Conditional Memory in LLMs

Tianqi Chen, Xunchao Li, Linchuan Xie, Guanghua Yu, Jianchen Zhu, Peisong Wang, Jian ChengarXiv 2026 · arXiv

TL;DR

GateRQ is a novel post-training quantization method for LLMs that use large "conditional memory" tables (like DeepSeek’s Engram). By using the model’s internal gating mechanism to dynamically decide which lookups require high precision on-the-fly, GateRQ reduces GPU memory usage by nearly 8x and boosts throughput by up to 31% with negligible impact on factual accuracy.

What Problem Does This Paper Address?

Modern LLMs, such as DeepSeek-V4.1 and Qwen, have begun offloading factual knowledge into "conditional memory" layers—massive hashed n-gram tables that sit alongside the standard Transformer backbone. While these tables are computationally cheap, they are physically enormous. For DeepSeek-V4.1-Flash, these tables occupy over 200 GB, consuming 40% of the model's total parameters.

The challenge lies in compressing these tables without losing the "long-tail" facts they are designed to store. Standard quantization techniques often fail here. Surprisingly, the common wisdom of keeping frequently accessed rows at higher precision actually backfires in this context. The authors found that refining lookups based on access frequency is actually worse than refining them at random.

Methods

The authors introduce GateRQ (Gate-driven Residual Quantization), a post-training scheme that treats precision as a dynamic property of a specific access rather than a static property of a stored row.

  1. Residual Ladder: Each row in the memory table is stored as a "ladder" of 1-bit residual codes. Level 1 is always kept in fast GPU memory, while deeper levels (Levels 2 and 3) are stored in host (CPU) memory.
  2. Gate-Driven Selection: When a token retrieves a row, it first reads the 1-bit Level 1 code from the GPU. The model uses this to compute a "gate" value—a scalar that determines how much this specific memory lookup will influence the final output.
  3. Dynamic Precision: If the gate value is high (indicating an important lookup), the system dynamically fetches the deeper residual levels from host memory via PCIe to refine the value. If the gate is low, it sticks with the 1-bit approximation.
The GateRQ architecture: Level 1 stays in GPU memory to compute the gate, which then decides whether to fetch deeper levels from host memory.

Datasets

The researchers evaluated GateRQ across two models (DeepSeek-V4.1-Flash and Qwen3.8-Flash-Next) using several key benchmarks:

  • PopQA: A benchmark specifically designed for factual questions about "long-tail" (rare) entities.
  • TriviaQA & NQ-open: Standard open-domain question-answering tasks.
  • CMMLU: A comprehensive Chinese multitask knowledge evaluation.
  • RedPajama: Used as a counting corpus to analyze n-gram access frequencies.

Key Insights and Ideas

  • The Gate is the Weight: The quantization error of a retrieved row is directly scaled by its gate. A large gate amplifies error; a small gate suppresses it.
  • Importance vs. Frequency: In conditional memory, importance runs against frequency. Common words (like "the") are accessed constantly but have tiny gate values because they carry little factual information. Rare entities (like a specific book title) are accessed seldomly but have large gates when they appear.
Example of a PopQA question where rare entity tokens receive much higher gate values than frequent function words.
  • Context Matters: The importance of a row isn't just about the row itself; it's about the context. The same n-gram might be critical in one sentence and irrelevant in another. GateRQ handles this by making precision decisions per lookup.
  • 1-Bit Gates are Enough: A crucial finding is that the gating mechanism is remarkably robust to quantization. Gates computed from a 1-bit representation correlate almost perfectly ($r \geq 0.989$) with those from the full FP8 table. This allows the model to "decide" which data is important before it even sees the high-precision data.

Key Results

  • Memory Efficiency: GateRQ cut GPU table memory for DeepSeek-V4.1-Flash from 23.6 GiB to 3.0 GiB per GPU (a 7.8x reduction). This freed up enough space to enlarge the KV cache by 8.1x.
  • Throughput Gains: By reducing GPU memory pressure and optimizing data transfer, GateRQ increased vLLM throughput by 11% to 31% compared to using FP8 tables.
  • Accuracy Preservation: On the challenging PopQA benchmark, GateRQ (at 2.10 bits per lookup) stayed within 0.2 points of the original FP8 model's accuracy.
  • Superiority over Baselines: In 36 paired comparisons, gate-based selection consistently outperformed selection based on access frequency, expected gate (prior), or random choice.
PopQA accuracy vs. bits per lookup, showing GateRQ consistently outperforming other selection rules across different quantization codecs.

Limitations and Future Work

  • Latency Trade-offs: While GateRQ is highly effective for large prefill blocks, the fixed latency of host memory reads (via PCIe) means the benefits are less pronounced for very small batch sizes or short sequences.
  • Implementation Scope: The current native implementation is a patch for vLLM specifically targeting DeepSeek's architecture. Generalizing this to a wider variety of conditional memory implementations remains future work.
  • Retraining Potential: GateRQ is a post-training method. The authors suggest that incorporating gate-aware objectives during the initial training or fine-tuning of these memory layers could yield even better compression ratios.

Summary updated

Abstract

Conditional memory layers, such as DeepSeek's Engram, attach large hashed n-gram tables to a language model, and removing them from DeepSeek-V4.1-Flash costs 36.8 PopQA points. Low-bit quantization loses long-tail facts first, and the standard remedy, giving more bits to frequently accessed rows, backfires: refining lookups by access frequency is worse than refining them at random. We trace this to the gate that scales each retrieved row. A lookup's quantization error is weighted by its gate, which is small for frequent n-grams, large for the rare n-grams of long-tail entities, and computable from a 1-bit code. GateRQ therefore stores each row as a ladder of 1-bit residual codes, keeps level 1 in GPU memory to compute the gate, and reads deeper levels only for the lookups the gate ranks highest. Without retraining, GateRQ cuts GPU table memory by 7.8x on DeepSeek-V4.1-Flash, which raises vLLM throughput by 11%-31% over FP8 tables, while losing about 1 PopQA point at 1.50 bits read per lookup and 0.2 points at 2.10 bits across two models.
BibTeX
@misc{chen2026let,
  author = {Tianqi Chen and Xunchao Li and Linchuan Xie and Guanghua Yu and Jianchen Zhu and Peisong Wang and Jian Cheng},
  title = {{Let the Gate Decide: Per-Lookup Precision for Quantized Conditional Memory in LLMs}},
  year = {2026},
  eprint = {2610.35985},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.35985},
}
Figure