AscMLA: Unlocking FP4 Efficiency for Long-Context MLA Decoding through Asymmetric Scaling

Zechun Zhou, Zhiyong Zheng, Xiaomin Zhang, Lin Ye, Han Li, Jingwei Sun, Guangzhong Sun, Jing LiarXiv 2026 · arXiv

TL;DR

AscMLA introduces a novel representation and kernel co-design that enables efficient FP4 quantization for Multi-Head Latent Attention (MLA). By using asymmetric scaling—fitting different scales for K and V to a single shared FP4 payload—it achieves a 38.2% reduction in KV cache memory and up to a 1.35× speedup over state-of-the-art FP8 implementations without sacrificing significant accuracy.

What Problem Does This Paper Address?

As Large Language Models (LLMs) move toward longer context windows, the KV cache becomes a massive bottleneck in both memory capacity and bandwidth. While moving from 8-bit (FP8) to 4-bit (FP4) precision seems like an obvious fix, it introduces a specific technical headache for Multi-Head Latent Attention (MLA), a popular architecture used in models like DeepSeek-V2/V3.

MLA saves space by sharing a compressed "latent" representation for both Keys (K) and Values (V). However, FP4 has a very narrow dynamic range and requires "block scaling" (grouping values and giving them a shared scale factor) to maintain accuracy. The problem is a structural mismatch:

  1. Query-Key (QK) multiplication reduces over latent channels, requiring scales aligned with those channels.
  2. Probability-Value (PV) multiplication reduces over tokens, requiring scales aligned with tokens.

Existing methods either duplicate the latent (losing the storage benefit) or perform expensive dequantization during the decoding process, which eats up the performance gains.

Fig. 1. NVFP4 scale-group mismatch in MLA. (a) The 16 channels in the blue row share one cached scale, which native QK MMA can use directly. (b) The 16 tokens in the orange column have separate cached scales, whereas native PV MMA requires one shared scale for the entire group.

Methods

The authors propose AscMLA, which uses a "representation and kernel co-design" to handle this mismatch directly in the FP4 domain.

1. Asymmetric K/V Scaling

Instead of trying to find one set of scales that works for both K and V, AscMLA uses Asymmetric Scaling. It first quantizes the latent for the V-view using one scale per token. Then, keeping those FP4 codes fixed, it "fits" a second set of scales for the K-view (one scale per 16-channel block). This allows a single FP4 payload to serve two different mathematical masters.

2. Kernel Co-design: "Folding" Scales

To avoid dequantizing V back to FP8, the kernel "folds" the per-token V-scales directly into the softmax probabilities. This mathematical trick allows the final PV multiplication to use the raw FP4 values directly. The kernel utilizes native hardware instructions: NVFP4 MMA for the QK product and MXFP8–MXFP4 MMA for the PV product.

Fig. 2. Overview of AscMLA. PV-first quantization produces a shared FP4 payload Z and per-token V scales a. QK-scale fitting derives block K scales k without changing Z. The payload is loaded into K/V SMEM layouts ZK and ZV for native NVFP4 QK and MXFP8–MXFP4 PV. V scales are folded into probabilities before PV.

3. Pipelined Scale Delivery

To keep the GPU's asynchronous pipeline full, the authors implemented consumer-specific scale delivery. They use K-scale multibuffering to prevent "wait-before-overwrite" stalls and V-scale multistage prefetching to overlap the loading of scales with the calculation of attention scores.

Fig. 3. Consumer-specific scale delivery. Four K-scale TMEM slots remove overwrite waits between QK MMAs. V scales reach registers before waiting for scores, freeing SMEM slots for prefetching while register copies feed probability conversion. Here ai is the token-scale vector for KV tile i. REG denotes registers.

Datasets

The authors evaluated AscMLA using:

  • Synthetic Benchmarks: 40 different workload configurations (varying batch sizes and context lengths) using Gaussian, outlier-heavy, and correlated distributions to test robustness.
  • Downstream Accuracy: Evaluated on GPQA-Diamond using the Kimi-K2.6 and Kimi-K3 models.
  • Serving Throughput: Real-world serving scenarios using the ShareGPT dataset.

Key Insights and Ideas

  • Sharing Codes $\neq$ Sharing Scales: The "aha moment" is realizing that the storage-heavy part of the KV cache is the 4-bit codes. You can afford to store two different sets of small scale factors (one for K, one for V) if it means you can keep the main payload in 4-bit and process it natively.
  • Softmax Folding: By treating the V-scale as a modifier to the attention probability rather than the value itself, the authors bypassed the need for a separate dequantization kernel, which was the primary bottleneck in previous FP4 MLA attempts like CAKE.
  • Hardware Alignment: The design specifically targets the architectural strengths of modern GPUs (like the Blackwell/Hopper generation's TMA and TMEM), ensuring that the overhead of managing asymmetric scales doesn't outweigh the bandwidth savings.

Key Results

  • Memory Savings: Uses 38.2% less memory than FP8 KV caches.
  • Speedups: Achieves a geometric-mean speedup of 1.35× over TRT-LLM’s FP8 kernels and 1.26× over CAKE’s NVFP4 MLA kernels.
  • Serving Throughput: In end-to-end testing with Kimi-K2.6, AscMLA delivered 18.8% higher throughput than the FP8 baseline while still meeting strict latency targets (SLOs).
  • Accuracy: Maintained high fidelity, scoring 89.4% on GPQA-Diamond (compared to the 90.5% FP8 baseline), a negligible trade-off for the massive efficiency gains.

Limitations and Future Work

  • Outlier Sensitivity: Because AscMLA uses a single scale for an entire token's V-view, a few extremely large "outlier" values can reduce the precision for the rest of the values in that token.
  • Hardware Specificity: The current implementation is highly optimized for NVIDIA's latest architectures (using specific MMA instructions). Porting these benefits to other hardware (like AMD or specialized AI accelerators) would require significant re-engineering of the scale-folding logic.
  • Future Work: The authors suggest exploring even lower bit-widths or more granular scaling (like 2-bit quantization) using similar asymmetric strategies.

Summary updated

Abstract

Long-context decoding repeatedly reads a growing KV cache. FP4 reduces cache storage and traffic but needs block scaling to handle its limited dynamic range. Applying FP4 to multi-head latent attention (MLA) requires reconciling different K/V scaling requirements on a shared latent. Existing approaches address this mismatch through dequantization or separate encodings, adding execution or storage overhead. We present AscMLA, which combines asymmetric K/V scaling with pipelined scale delivery to process a single FP4 payload directly. AscMLA uses 38.2% less cache memory than FP8 and achieves geometric-mean speedups of 1.35x over TRT-LLM's FP8 and 1.26x over CAKE's NVFP4 MLA kernels.
BibTeX
@misc{zhou2026ascmla,
  author = {Zechun Zhou and Zhiyong Zheng and Xiaomin Zhang and Lin Ye and Han Li and Jingwei Sun and Guangzhong Sun and Jing Li},
  title = {{AscMLA: Unlocking FP4 Efficiency for Long-Context MLA Decoding through Asymmetric Scaling}},
  year = {2026},
  eprint = {2611.26106},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.26106},
}

Loading related content...

Figure