CALFA-KV: Joint Calibration and Fisher-Guided Anchors for 1-Bit KV Cache Compression

Xiancai Tian, Leichao Zhang, Shenhui Zhang, Yi Shan, Hailong Li, Feng Lin, Xiaotian Yu, Aijun LiarXiv 2026 · arXiv

TL;DR

CALFA-KV achieves high-fidelity 1-bit Key-Value (KV) cache compression by combining offline joint calibration with an online "anchor" system. By retaining a tiny fraction of the most sensitive tokens in high precision—guided by Fisher information—it narrows the performance gap to uncompressed models to within 0.9–2.0 points on major benchmarks while using only ~1.25 bits per channel.

What Problem Does This Paper Address?

Large Language Models (LLMs) are notorious memory hogs, and a significant portion of that memory goes into the KV cache—the stored intermediate states of previous tokens used to generate the next one. As sequence lengths grow, the KV cache can easily exceed the available GPU memory, limiting batch sizes and context lengths.

While 1-bit quantization (compressing each value to a single bit) offers a massive 16x theoretical reduction in memory compared to FP16, it usually destroys model accuracy. The fundamental challenge is that not all tokens or channels are created equal: some are highly sensitive to noise, while others are robust. Existing methods either try to find better quantization levels or keep a few "important" tokens in high precision, but they often fail to account for how quantization errors propagate through the attention mechanism to the final output.

Methods

The authors propose CALFA-KV (CALibration and Fisher-guided Anchors), which operates in two phases:

  1. Offline Joint Calibration: Instead of just minimizing the reconstruction error of the K and V vectors in isolation, CALFA-KV jointly fits a key center, radial normalization parameters, and vector codebooks to match the projected attention output of the original frozen model. This ensures the quantizer is optimized for what actually matters: the final result of the attention layer.
  2. Fisher-Guided Anchor Selection: During inference, the system maintains a small set of "anchors"—tokens kept in full FP16 precision. To decide which tokens to save, the authors estimate channel-wise Fisher-type sensitivities offline. These sensitivities act as weights: a token is prioritized for high-precision storage if it has a large reconstruction error in a channel that the model is particularly sensitive to.
Fig. 1. CALFA-KV for 1-bit KV cache compression. (a) Offline joint calibration proposes updates with fixed codeword assignments and tests them using deployment-precision replay after re-encoding. Channel sensitivities are estimated separately from reference-loss gradients. (b) During inference, sensitivity- weighted reconstruction errors rank historical KV states for FP16 retention, with up to N anchors per layer and KV head. Attention combines decoded history, restored anchors, and unquantized recent and live K/V. The nominal 1-bit rate refers to codeword indices.

The system uses a streaming "Top-N" selection process to efficiently update the anchor set as new tokens are generated, ensuring the most critical historical information is always preserved at high fidelity.

Datasets

The researchers evaluated CALFA-KV across three different models (Qwen3-4B, Llama-3.1-8B, and Qwen3-30B) using a diverse suite of seven benchmarks:

  • GSM8K & MATH500: Mathematical reasoning.
  • IFEval: Instruction following.
  • HumanEval+ & LiveCodeBench: Code generation.
  • LongBench: Long-context understanding.
  • GPQA-Diamond: Expert-level scientific question answering.

Key Insights and Ideas

  • Sensitivity > Magnitude: A large quantization error in a "quiet" channel might not matter, while a tiny error in a "sensitive" channel can derail the model. Fisher information provides a mathematically grounded way to identify these sensitive spots.
  • Downstream Optimization: Calibrating a quantizer to minimize local vector error is a sub-optimal proxy. By calibrating against the attention output, the authors account for how the softmax operation and value-weighting amplify or suppress specific errors.
  • The Power of a Few: You don't need many anchors. With a budget of just 56 FP16 anchors per layer/head, the model recovers nearly all the performance lost to 1-bit quantization.
  • Fisher-Guided vs. Unweighted: Simply keeping tokens with the highest reconstruction error (unweighted) is significantly less effective than weighting those errors by Fisher sensitivity.

Key Results

CALFA-KV sets a new state-of-the-art for ultra-low-bit KV cache compression:

  • Near-Lossless Performance: The d8b10 configuration (1.25 bits per channel + anchors) achieved mean scores across seven benchmarks that were only 0.90 to 2.01 points below the BF16 baseline.
  • Outperforming Baselines: At a 1-bit rate, CALFA-KV outperformed existing methods like KIVI, OSCAR, and TaSQ. For example, on the Qwen3-4B model, it beat the TaSQ baseline by 4.43 points on average.
  • Error Reduction: Layerwise analysis showed that Fisher-guided selection reduced hidden-state relative Mean Squared Error (MSE) by 32.2% compared to using no anchors.
Layerwise hidden-state error comparison showing Fisher-guided selection (red) consistently outperforming random and unweighted baselines.

Limitations and Future Work

  • Model-Specific Calibration: The calibration and sensitivity estimation must be performed offline for each specific model, which adds a pre-processing step.
  • Fixed Anchor Budget: The current implementation uses a fixed per-layer, per-head anchor budget. The authors suggest that dynamic budgets (allocating more anchors to more sensitive layers) could be a promising future direction.
  • Hardware Integration: While the paper focuses on the algorithmic framework and accuracy, the actual speedup in a production environment would depend on specialized kernels that can efficiently handle the hybrid 1-bit/FP16 attention computation.

Summary updated

Abstract

1-bit key-value (KV) cache compression can combine quantizer calibration with selective high-precision storage to reduce reconstruction error. We propose Calibration and Fisher-guided Anchors for KV cache compression (CALFA-KV). Offline, we jointly fit a key center, radial normalization parameters, and K/V vector codebooks to match the frozen model's projected attention output. We also estimate channel-wise Fisher-type sensitivities from squared gradients of a reference next-token cross-entropy loss. Online, we use these sensitivities to weight the squared reconstruction errors of each KV state. Here, a KV state is one token's key-value pair within one layer and KV head. We retain the highest-scoring states at high precision under a per-layer, per-KV-head budget. Under exact restoration, we bound the excess risk of score-based anchor selection relative to the optimum at the same anchor budget, where risk is the conditional mean square of the first-order loss change. With a budget of 56 additional FP16 anchors per layer and KV head, CALFA-KV (d8b8) outperforms all evaluated nominal 1-bit baselines on the mean score across seven benchmarks for each of the three models. At the same anchor budget, CALFA-KV (d8b10) uses a nominal index rate of 1.25 bits per channel and narrows the gap to BF16 to 0.90--2.01 points on the same metric.
BibTeX
@misc{tian2026calfa,
  author = {Xiancai Tian and Leichao Zhang and Shenhui Zhang and Yi Shan and Hailong Li and Feng Lin and Xiaotian Yu and Aijun Li},
  title = {{CALFA-KV: Joint Calibration and Fisher-Guided Anchors for 1-Bit KV Cache Compression}},
  year = {2026},
  eprint = {2611.10511},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.10511},
}
Figure