Abstract
Existing key-value (KV) cache compression methods for large language models (LLMs) often rely on token eviction, which risks losing critical local information in both long prefilling and decoding scenarios. When extrapolating beyond the pretrained context length, their performance degrades sharply on long-context benchmarks. Motivated by the observation in the frequency domain that the context information is concentrated in the low-frequency components, we propose FreqKV, a parameter-free and architecture-agnostic approach. It iteratively compresses the increasing KV cache in the frequency domain, allowing models to process lengthy contexts efficiently. With minimal training at 8K length, FreqKV extends the context window of LLaMA-2-7B up to 256K tokens while maintaining stable perplexity. Extensive experiments across prefilling and decoding demonstrate that FreqKV enables robust context window extension and consistently outperforms existing KV cache compression methods on LLaMA-2 and LLaMA-3.
Introduction
Large Language Models (LLMs) are increasingly required to process massive sequences, yet they remain computationally expensive and often "forgetful" when dealing with long contexts. Their performance is typically constrained by a pre-defined context window established during training. To address this, researchers have explored two primary strategies: position encoding extension (e.g., RoPE interpolation) and KV cache compression.
While position encoding methods suffer from quadratic computational costs, current KV cache compression techniques often rely on token eviction—discarding "unimportant" tokens. This approach leads to permanent information loss and poor extrapolation performance far beyond the training length. FreqKV addresses these limitations by shifting from token-domain eviction to frequency-domain summarization, leveraging signal processing techniques to retain global semantic structure.
Methodology: Compression in the Frequency Domain
The core innovation of FreqKV is performing compression in the frequency domain rather than the time (token) domain. This approach is built on several key components:
- Discrete Cosine Transform (DCT): FreqKV applies DCT to the KV cache along the sequence dimension, transforming the sequence of tokens into a set of frequency components.
- Low-Pass Filtering: Observations of LLM hidden states reveal a "Frequency Principle": the energy and critical information of KV states are concentrated in low-frequency components. FreqKV retains these low-frequency parts while discarding high-frequency components, which often represent fine-grained local noise.
- Iterative Compression: As generation progresses, the cache is compressed iteratively. Older tokens undergo more compression cycles, while recent tokens remain at high fidelity. This creates a natural "sliding window" effect where the present is high-resolution and the distant past is a compressed summary.
- Rescaling and Attention Sinks: To maintain numerical stability, a rescaling factor is introduced after the Inverse DCT (IDCT). Additionally, "attention sinks" (the initial tokens of a sequence) are kept uncompressed to ensure model stability.
The Frequency Principle in LLMs
The effectiveness of FreqKV stems from the inherent bias of LLMs toward low-frequency information. Just as JPEG compression removes high-frequency details from images with minimal perceptual loss, FreqKV removes high-frequency "noise" from the KV cache. This bias is observed consistently across different layers of the model.
As shown in the frequency analysis, the magnitude of components drops off sharply after the initial low-frequency indices. This concentration persists even in the deepest layers of the network, justifying the use of low-pass filtering as a global compression strategy.
Experimental Results
FreqKV was evaluated across a wide range of benchmarks, including PG-19 for language modeling, LongBench for understanding, and the Needle-in-a-Haystack test for retrieval.
- Massive Scaling: FreqKV successfully extended LLaMA-2-7B (trained on 8K tokens) to a 256K token window—a 64x extension—while maintaining stable perplexity.
- Superior Retrieval and Understanding: On the RULER benchmark, FreqKV significantly outperformed existing compression methods like SnapKV and PyramidKV. While baseline methods often collapsed at 16K evaluation lengths, FreqKV remained robust.
- Efficiency and Overhead: The computational cost of DCT/IDCT is negligible, accounting for less than 0.3% of total decoding time. Furthermore, FreqKV reduced memory requirements by approximately 44% and training time by 40% compared to full-cache fine-tuning.
Conclusion and Future Work
FreqKV provides a parameter-free, architecture-agnostic solution for context window extension. By leveraging the frequency-domain characteristics of LLMs, it achieves massive scaling with minimal computational overhead. While the method shows a slight performance gap in symbolic tasks (like those in the Proof-pile dataset) due to the loss of high-frequency details, it represents a significant step forward in efficient long-context processing. Future work will explore more sophisticated frequency-domain filters to better preserve technical and symbolic information.
Summary updated