TL;DR
CALFA-KV achieves high-fidelity 1-bit Key-Value (KV) cache compression by combining offline joint calibration with an online "anchor" system. By retaining a tiny fraction of the most sensitive tokens in high precision—guided by Fisher information—it narrows the performance gap to uncompressed models to within 0.9–2.0 points on major benchmarks while using only ~1.25 bits per channel.
What Problem Does This Paper Address?
Large Language Models (LLMs) are notorious memory hogs, and a significant portion of that memory goes into the KV cache—the stored intermediate states of previous tokens used to generate the next one. As sequence lengths grow, the KV cache can easily exceed the available GPU memory, limiting batch sizes and context lengths.
While 1-bit quantization (compressing each value to a single bit) offers a massive 16x theoretical reduction in memory compared to FP16, it usually destroys model accuracy. The fundamental challenge is that not all tokens or channels are created equal: some are highly sensitive to noise, while others are robust. Existing methods either try to find better quantization levels or keep a few "important" tokens in high precision, but they often fail to account for how quantization errors propagate through the attention mechanism to the final output.
Methods
The authors propose CALFA-KV (CALibration and Fisher-guided Anchors), which operates in two phases:
- Offline Joint Calibration: Instead of just minimizing the reconstruction error of the K and V vectors in isolation, CALFA-KV jointly fits a key center, radial normalization parameters, and vector codebooks to match the projected attention output of the original frozen model. This ensures the quantizer is optimized for what actually matters: the final result of the attention layer.
- Fisher-Guided Anchor Selection: During inference, the system maintains a small set of "anchors"—tokens kept in full FP16 precision. To decide which tokens to save, the authors estimate channel-wise Fisher-type sensitivities offline. These sensitivities act as weights: a token is prioritized for high-precision storage if it has a large reconstruction error in a channel that the model is particularly sensitive to.
The system uses a streaming "Top-N" selection process to efficiently update the anchor set as new tokens are generated, ensuring the most critical historical information is always preserved at high fidelity.
Datasets
The researchers evaluated CALFA-KV across three different models (Qwen3-4B, Llama-3.1-8B, and Qwen3-30B) using a diverse suite of seven benchmarks:
- GSM8K & MATH500: Mathematical reasoning.
- IFEval: Instruction following.
- HumanEval+ & LiveCodeBench: Code generation.
- LongBench: Long-context understanding.
- GPQA-Diamond: Expert-level scientific question answering.
Key Insights and Ideas
- Sensitivity > Magnitude: A large quantization error in a "quiet" channel might not matter, while a tiny error in a "sensitive" channel can derail the model. Fisher information provides a mathematically grounded way to identify these sensitive spots.
- Downstream Optimization: Calibrating a quantizer to minimize local vector error is a sub-optimal proxy. By calibrating against the attention output, the authors account for how the softmax operation and value-weighting amplify or suppress specific errors.
- The Power of a Few: You don't need many anchors. With a budget of just 56 FP16 anchors per layer/head, the model recovers nearly all the performance lost to 1-bit quantization.
- Fisher-Guided vs. Unweighted: Simply keeping tokens with the highest reconstruction error (unweighted) is significantly less effective than weighting those errors by Fisher sensitivity.
Key Results
CALFA-KV sets a new state-of-the-art for ultra-low-bit KV cache compression:
- Near-Lossless Performance: The
d8b10configuration (1.25 bits per channel + anchors) achieved mean scores across seven benchmarks that were only 0.90 to 2.01 points below the BF16 baseline. - Outperforming Baselines: At a 1-bit rate, CALFA-KV outperformed existing methods like KIVI, OSCAR, and TaSQ. For example, on the Qwen3-4B model, it beat the TaSQ baseline by 4.43 points on average.
- Error Reduction: Layerwise analysis showed that Fisher-guided selection reduced hidden-state relative Mean Squared Error (MSE) by 32.2% compared to using no anchors.
Limitations and Future Work
- Model-Specific Calibration: The calibration and sensitivity estimation must be performed offline for each specific model, which adds a pre-processing step.
- Fixed Anchor Budget: The current implementation uses a fixed per-layer, per-head anchor budget. The authors suggest that dynamic budgets (allocating more anchors to more sensitive layers) could be a promising future direction.
- Hardware Integration: While the paper focuses on the algorithmic framework and accuracy, the actual speedup in a production environment would depend on specialized kernels that can efficiently handle the hybrid 1-bit/FP16 attention computation.
Summary updated