Cocktail: Chunk-Adaptive Mixed-Precision Quantization for Long-Context LLM Inference

Wei Tao, Bin Zhang, Xiaoyang Qu, Jiguang Wan, Jianzong Wang

Abstract

As Large Language Models (LLMs) expand toward context windows of 128K tokens and beyond, the memory requirements for storing Key-Value (KV) caches have become a primary bottleneck for inference. Traditional quantization methods often sacrifice model accuracy for compression, while existing mixed-precision approaches suffer from high computational overhead and hardware inefficiency. This paper introduces Cocktail, a framework that utilizes chunk-adaptive mixed-precision quantization to optimize long-context inference. By identifying relevant context segments and reordering them in memory for hardware alignment, Cocktail reduces GPU memory usage by up to 42% and improves inference speed by over 50% with negligible impact on accuracy.

Introduction

The transition toward "infinite" context windows is a defining trend in LLM development, enabling applications in long-document analysis, legal review, and complex code generation. However, this progress is hindered by the KV cache—the storage of previous tokens' keys and values to avoid redundant computation. For a model like Llama2-13B with a 128K context, the KV cache can exceed 100GB, surpassing the capacity of a single NVIDIA A100 GPU.

While uniform quantization (e.g., compressing data to 4-bit or 2-bit) can mitigate memory pressure, it often degrades the model's reasoning capabilities. Mixed-precision methods, which preserve critical tokens in high resolution, offer a potential solution but are historically difficult to implement efficiently. Token-level importance search is computationally expensive during live inference, and scattered bit-widths in memory lead to hardware underutilization. Cocktail addresses these challenges through a hardware-aware, chunk-based approach.

The Cocktail Framework

Cocktail introduces two primary innovations to make mixed-precision quantization viable for production environments: Chunk-level Quantization Search and Chunk-level KV Cache Computation.

Rather than evaluating the importance of every individual token, Cocktail partitions the context into discrete chunks (e.g., 32 tokens). It employs a lightweight retrieval model, such as Facebook-Contriever, to calculate the semantic similarity between the user’s query and each context chunk. Based on this similarity, chunks are assigned different precision levels:

  • High Similarity: Retained in FP16 to preserve critical information.
  • Medium Similarity: Compressed to INT4 for balanced efficiency.
  • Low Similarity: Compressed to INT2 to maximize memory savings.

This granularity provides a "sweet spot"—it is coarse enough to make the search overhead negligible but fine enough to capture the relevant semantic clusters within a document.

2. Chunk-level KV Cache Computation

To solve the "hardware tax" associated with mixed-precision data, Cocktail reorders chunks in physical memory. Standard GPUs struggle when different bit-widths are interleaved, as it breaks the efficiency of Single Instruction, Multiple Data (SIMD) operations. Cocktail groups all INT2, INT4, and FP16 chunks into contiguous blocks. This allows the hardware to perform optimized cache-line reads and high-speed computation. The results are then mathematically aggregated to ensure the final output remains consistent with the original sequence order.

Experimental Evaluation

Methodology

The researchers evaluated Cocktail using the LongBench benchmark, a comprehensive suite designed for long-context tasks. The evaluation covered:

  • Single-Document QA: Qasper
  • Summarization: QMSum, MultiNews, SAMSum
  • Few-shot Learning: TREC, TriviaQA
  • Code Completion: LCC, RepoBench-P

Tests were conducted across four prominent models: Llama2-7B, Llama2-13B, Mistral-7B, and Longchat-7B.

Key Results

  • Accuracy Preservation: Cocktail achieved the highest average performance among state-of-the-art (SOTA) methods, including Atom, KIVI, and KVQuant. On average, it dropped only 0.055 points compared to the uncompressed FP16 baseline.
  • Memory and Latency: The framework reduced GPU memory usage by 12% to 42% compared to standard FP16 models. Furthermore, the Time Per Output Token (TPOT) was reduced by 32% to 52%.
  • Scalability: While the initial search adds minor latency at small batch sizes, Cocktail’s throughput scales superiorly as batch sizes increase, outperforming token-level alternatives that struggle with search complexity.

Discussion and Insights

The Sparsity of Relevance

A core insight of the study is that for any specific query, only a small fraction of a massive document is truly relevant. This inherent sparsity justifies aggressive 2-bit quantization for the majority of the context, provided the "needles in the haystack" are kept at high precision.

Memory Layout as a Performance Driver

The study emphasizes that software-level compression is insufficient without hardware-level consideration. By aligning the memory layout with the quantization strategy, Cocktail avoids the performance degradation typically seen in mixed-precision systems, proving that memory management is as critical as the compression algorithm itself.

Limitations and Future Work

Despite its success, Cocktail has areas for refinement. The system is sensitive to chunk size; if chunks are too large (e.g., 256 tokens), important information may be diluted and incorrectly quantized to low precision. Additionally, the framework's effectiveness is tied to the performance of the external encoder (Contriever). Future research will focus on automating the similarity thresholds (α and β) to eliminate manual tuning and exploring tighter integration between the retrieval model and the LLM backbone.

Summary updated

Abstract

Recently, large language models (LLMs) have been able to handle longer and longer contexts. However, a context that is too long may cause intolerant inference latency and GPU memory usage. Existing methods propose mixed-precision quantization to the key-value (KV) cache in LLMs based on token granularity, which is time-consuming in the search process and hardware inefficient during computation. This paper introduces a novel approach called Cocktail, which employs chunk-adaptive mixed-precision quantization to optimize the KV cache. Cocktail consists of two modules: chunk-level quantization search and chunk-level KV cache computation. Chunk-level quantization search determines the optimal bitwidth configuration of the KV cache chunks quickly based on the similarity scores between the corresponding context chunks and the query, maintaining the model accuracy. Furthermore, chunk-level KV cache computation reorders the KV cache chunks before quantization, avoiding the hardware inefficiency caused by mixed-precision quantization in inference computation. Extensive experiments demonstrate that Cocktail outperforms state-of-the-art KV cache quantization methods on various models and datasets. Our code is presented on https://github.com/Sullivan12138/Cocktail.
BibTeX
@inproceedings{tao2025cocktail,
  author = {Wei Tao and Bin Zhang and Xiaoyang Qu and Jiguang Wan and Jianzong Wang},
  title = {{Cocktail: Chunk-Adaptive Mixed-Precision Quantization for Long-Context LLM Inference}},
  booktitle = {DATE},
  year = {2025},
  url = {https://openreview.net/forum?id=E2JftQuXk8},
}
Figure