Fused Kernels for Quantized Execution in Large Models

This direction is split into other directions: fused quantization kernels. Its last version stays readable and citable here.

Version 1 of 1123 papers144 conceptsfrozen 2026-10-04

Quantization is often framed as a compression problem: how can we represent a 16-bit weight in 4 bits without losing the model's "knowledge"? However, for modern Large Language Models (LLMs), the bottleneck isn't just storage—it's the speed at which data moves from memory to the processor. If a system must dequantize data back to high precision before performing a calculation, the overhead of that conversion often negates the benefits of the smaller data size.

The research direction emerging across these papers is a shift from quantization for storage to quantization for execution. By fusing dequantization, scaling, and mathematical transforms directly into the core computational kernels, researchers are enabling models to run at the theoretical speeds promised by low-precision hardware.

The Memory-Compute Bottleneck

In standard inference, the GPU spends most of its time waiting for weights to arrive from global memory (VRAM) to the registers where math happens. Quantization helps by reducing the amount of data that needs to move. However, as Chuang Gan and Song Han point out in AWQ, if the processor has to perform a separate "unpacking" step to turn those 4-bit values back into 16-bit floats, the extra instructions can actually slow down the model.

The solution is kernel fusion. Instead of treating dequantization as a pre-processing step, researchers like Young Jin Kim in FineQuant integrate it directly into the matrix multiplication (GEMM) kernel. By performing dequantization "on-the-fly" as weights are loaded into registers, the computational cost is "hidden" behind the time it takes to move the data.

From Weight-Only to Quantized Arithmetic

While weight-only quantization (W4A16) speeds up memory-bound decoding, it doesn't help with compute-bound phases like the initial prompt processing (prefill). To accelerate the actual math, the system must use integer tensor cores.

Hao Kang and colleagues demonstrate this in TurboAttention, where they execute the attention mechanism directly on quantized data using integer operations. This avoids the latency of jumping back and forth between integer and floating-point formats. Similarly, Ce Zheng in INT-FlashAttention shows that by integrating quantization into the tiling loops of the FlashAttention algorithm, one can maintain the accuracy of the online softmax while reaping the throughput of 8-bit math.

Fusing the "Fixes" for Outliers

A major hurdle in low-bit quantization is the presence of "outliers"—specific channels or activations with much higher magnitudes that cause massive errors when squeezed into 4 bits. Recent work focuses on fusing the mathematical "fixes" for these outliers directly into the kernels to avoid performance penalties:

  1. Rotations and Smoothing: Tong Zhang and Junyang Lin in Rotated Runtime Smooth use Hadamard transforms to "spread out" outliers across many channels. By fusing these rotations into the GEMM kernel, they neutralize outliers with negligible overhead.
  2. Low-Rank Branches: Muyang Li and Song Han introduce SVDQuant, which peels off the most difficult outliers into a separate high-precision "low-rank" branch. Crucially, their Nunchaku engine fuses this dual-path execution into a single kernel, preventing the memory bottleneck of running two separate operations.
  3. Codebook Lookups: For extreme compression (2-bit), Mao Yang and the VPTQ team fuse vector quantization codebook lookups directly with matrix-vector multiplication, ensuring that even complex reconstruction logic doesn't stall the GPU.

Hardware-Software Co-Design

The most recent frontier involves tailoring these fused kernels to specific hardware architectures. Haoqian Meng and Yilun Luo demonstrate this in MicroMix and SageAttention3, which target the NVIDIA Blackwell architecture. By using specialized "microscaling" formats and FP4 Tensor Cores, they achieve massive throughput gains (over 1000 TOPS) by ensuring that data never leaves the fastest on-chip memory during the transition from storage to computation.

This trend is not limited to GPUs. Javier J. Poveda Rodrigo in V-Seek applies similar principles to RISC-V platforms, using vectorized kernels to perform dynamic quantization during execution. Whether on a server-class GPU or an edge device, the goal remains the same: eliminate the "seams" between data representation and data processing.

Papers in this version

Showing 60 of 123 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_fused_kernels_for_quantized_execution_in_b222eb76,
  title = {Fused Kernels for Quantized Execution in Large Models},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/fused-kernels-for-quantized-execution-in-large-models/rdr_b222eb76046d43d6ba72a0f88b7bab83},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_fused_kernels_for_quantized_execution_in_b222eb76_v1,
  title = {Fused Kernels for Quantized Execution in Large Models},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/fused-kernels-for-quantized-execution-in-large-models/rdr_b222eb76046d43d6ba72a0f88b7bab83/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}