Abstract
As Large Language Models (LLMs) expand toward context windows of 128K tokens and beyond, the memory requirements for storing Key-Value (KV) caches have become a primary bottleneck for inference. Traditional quantization methods often sacrifice model accuracy for compression, while existing mixed-precision approaches suffer from high computational overhead and hardware inefficiency. This paper introduces Cocktail, a framework that utilizes chunk-adaptive mixed-precision quantization to optimize long-context inference. By identifying relevant context segments and reordering them in memory for hardware alignment, Cocktail reduces GPU memory usage by up to 42% and improves inference speed by over 50% with negligible impact on accuracy.
Introduction
The transition toward "infinite" context windows is a defining trend in LLM development, enabling applications in long-document analysis, legal review, and complex code generation. However, this progress is hindered by the KV cache—the storage of previous tokens' keys and values to avoid redundant computation. For a model like Llama2-13B with a 128K context, the KV cache can exceed 100GB, surpassing the capacity of a single NVIDIA A100 GPU.
While uniform quantization (e.g., compressing data to 4-bit or 2-bit) can mitigate memory pressure, it often degrades the model's reasoning capabilities. Mixed-precision methods, which preserve critical tokens in high resolution, offer a potential solution but are historically difficult to implement efficiently. Token-level importance search is computationally expensive during live inference, and scattered bit-widths in memory lead to hardware underutilization. Cocktail addresses these challenges through a hardware-aware, chunk-based approach.
The Cocktail Framework
Cocktail introduces two primary innovations to make mixed-precision quantization viable for production environments: Chunk-level Quantization Search and Chunk-level KV Cache Computation.
1. Chunk-level Quantization Search
Rather than evaluating the importance of every individual token, Cocktail partitions the context into discrete chunks (e.g., 32 tokens). It employs a lightweight retrieval model, such as Facebook-Contriever, to calculate the semantic similarity between the user’s query and each context chunk. Based on this similarity, chunks are assigned different precision levels:
- High Similarity: Retained in FP16 to preserve critical information.
- Medium Similarity: Compressed to INT4 for balanced efficiency.
- Low Similarity: Compressed to INT2 to maximize memory savings.
This granularity provides a "sweet spot"—it is coarse enough to make the search overhead negligible but fine enough to capture the relevant semantic clusters within a document.
2. Chunk-level KV Cache Computation
To solve the "hardware tax" associated with mixed-precision data, Cocktail reorders chunks in physical memory. Standard GPUs struggle when different bit-widths are interleaved, as it breaks the efficiency of Single Instruction, Multiple Data (SIMD) operations. Cocktail groups all INT2, INT4, and FP16 chunks into contiguous blocks. This allows the hardware to perform optimized cache-line reads and high-speed computation. The results are then mathematically aggregated to ensure the final output remains consistent with the original sequence order.
Experimental Evaluation
Methodology
The researchers evaluated Cocktail using the LongBench benchmark, a comprehensive suite designed for long-context tasks. The evaluation covered:
- Single-Document QA: Qasper
- Summarization: QMSum, MultiNews, SAMSum
- Few-shot Learning: TREC, TriviaQA
- Code Completion: LCC, RepoBench-P
Tests were conducted across four prominent models: Llama2-7B, Llama2-13B, Mistral-7B, and Longchat-7B.
Key Results
- Accuracy Preservation: Cocktail achieved the highest average performance among state-of-the-art (SOTA) methods, including Atom, KIVI, and KVQuant. On average, it dropped only 0.055 points compared to the uncompressed FP16 baseline.
- Memory and Latency: The framework reduced GPU memory usage by 12% to 42% compared to standard FP16 models. Furthermore, the Time Per Output Token (TPOT) was reduced by 32% to 52%.
- Scalability: While the initial search adds minor latency at small batch sizes, Cocktail’s throughput scales superiorly as batch sizes increase, outperforming token-level alternatives that struggle with search complexity.
Discussion and Insights
The Sparsity of Relevance
A core insight of the study is that for any specific query, only a small fraction of a massive document is truly relevant. This inherent sparsity justifies aggressive 2-bit quantization for the majority of the context, provided the "needles in the haystack" are kept at high precision.
Memory Layout as a Performance Driver
The study emphasizes that software-level compression is insufficient without hardware-level consideration. By aligning the memory layout with the quantization strategy, Cocktail avoids the performance degradation typically seen in mixed-precision systems, proving that memory management is as critical as the compression algorithm itself.
Limitations and Future Work
Despite its success, Cocktail has areas for refinement. The system is sensitive to chunk size; if chunks are too large (e.g., 256 tokens), important information may be diluted and incorrectly quantized to low precision. Additionally, the framework's effectiveness is tied to the performance of the external encoder (Contriever). Future research will focus on automating the similarity thresholds (α and β) to eliminate manual tuning and exploring tighter integration between the retrieval model and the LLM backbone.
Summary updated