Beyond Bitwidth: Deployment-Constrained Mixed-Precision Quantization for Multilingual LLM Inference on NVIDIA Blackwell

Hyojung HanarXiv 2026 · arXiv

TL;DR

This paper moves beyond theoretical bitwidth to explore the "deployment feasibility" of quantized LLMs on NVIDIA Blackwell (B200) GPUs. By enforcing strict real-world constraints—memory limits, kernel compatibility, and multilingual quality gates—the authors demonstrate that while extreme low-bit formats like INT2 can execute natively, they often produce unusable text, and that a carefully tuned mixed-precision configuration (NVFP4 weights with INT8 protection) is required to maintain accuracy in languages like Korean.

What Problem Does This Paper Address?

Quantization research often focuses on a single metric: reducing bitwidth to save memory while minimizing the loss in average benchmark scores. However, a "compact" model on paper might fail in production for several reasons:

  1. Kernel Support: A format might not have an optimized GPU kernel, leading to slow "fallback" execution.
  2. Memory Overhead: The actual memory consumed when loading a model into a framework like vLLM can exceed the theoretical weight size.
  3. Language-Specific Degradation: A model that looks fine on English benchmarks might "collapse" in other languages, such as Korean.
  4. Non-Additive Effects: Protecting one "sensitive" layer might help, but the interactions between multiple quantized layers are complex and often unpredictable.

The authors tackle these gaps by studying Qwen3-8B on the NVIDIA B200 platform, seeking a configuration that satisfies memory, latency, and multilingual quality constraints simultaneously.

Methods

The researchers utilized the vLLM inference engine and its native support for various quantization kernels. They employed a "preregistered gate" approach, where a candidate configuration had to pass four strict conditions:

  • Execution: Must run through intended packed kernels for every module (no floating-point fallbacks).
  • Memory: Weight-loading memory must stay below 5.98 GiB.
  • Quality: Retention of the "teacher" (BF16) model's correctness must be $\ge 0.90$ across Korean, English, numerical, and reasoning tasks.
  • Latency: Time Per Output Token (TPOT) must not exceed 1.1x the BF16 reference.

They tested several families of quantization, as shown in the kernel selection table:

Table 1: Quantization families and the kernels vLLM 0.29.0 selected on B200.

To find the best mixed-precision setup, they used sensitivity diagnostics: restoring one module at a time to BF16 to see which ones provided the biggest accuracy boost, then greedily allocating a "protection" budget (using INT8) to those modules.

Datasets

The evaluation used a massive suite of benchmarks, including:

  • Multilingual/General: KMMLU, KoBEST (Korean), MMLU (English).
  • Reasoning/Knowledge: ARC, HellaSwag, BIG-bench Hard, MuSR.
  • Math: HRM8K (Korean math) and a seeded numerical problem generator.

Crucially, they used a held-out FINAL-81 set (5,168 items) that was only read once to verify the final deployment candidates, preventing "overfitting" to the development benchmarks.

Key Insights and Ideas

  • Native Execution $\neq$ Quality: Just because a kernel exists doesn't mean the output is usable. Both INT2 and INT3 executed natively on the B200, but INT2 models suffered from total "degeneration," either looping the same tokens or hitting output limits with gibberish.
Table 3: D80 INT2 grid diagnostics (diagnostic set, 4.49 GPU-hours). Retention is teacher-correct retention over all gate items; MC and Gen. split it into multiple-choice and generation items. Loop and Trunc. are the fractions of generation outputs that repeat or hit the output limit.
  • Sensitivity is Non-Additive: The benefit of restoring multiple modules is significantly less than the sum of their individual benefits. This suggests that "greedy" allocation strategies based on single-module sensitivity are a weak guide for complex mixed-precision mixtures.
  • Language Sensitivity Disagrees: Modules that are sensitive for reasoning tasks aren't necessarily the same ones sensitive for Korean language tasks. A single ranking cannot protect all capabilities.
  • The "Development Winner" Trap: A configuration that ranked first on the development set (D80) failed the quality gates on the independent final set. This underscores the need for rigorous, untouched evaluation sets in quantization research.

Key Results

The study identified a successful deployment candidate, A81, which used:

  • 98 NVFP4 weight-only modules.
  • 46 INT8 protected modules.
  • INT8 output head.
  • 16-bit activations (W4A16).

A81 passed all gates on the FINAL-81 set:

  • Memory: 5.917 GiB (under the 5.98 GiB limit).
  • Korean Retention: 0.904 (clearing the 0.90 gate).
  • Speed: TPOT ratio of 0.939 relative to BF16 (faster than the teacher).
  • Accuracy: It improved paired accuracy over the standard NVFP4 W4A4 baseline by +1.4 percentage points.

Limitations and Future Work

  • Attribution Ambiguity: The success of the A81 configuration comes from both moving to 16-bit activations and adding INT8 weight protection. The authors couldn't definitively isolate which change contributed more to the accuracy gain.
  • Single Model/Hardware: The results are specific to Qwen3-8B on NVIDIA B200. Different architectures or GPU generations might exhibit different sensitivity patterns.
  • Reproducibility: Final checkpoints were not retained, and the authors note that bitwise rebuild reproducibility remains unverified.
  • Future Directions: The authors suggest "causal ablation" on fresh held-out sets to better isolate the effects of activation precision versus weight protection, and exploring interaction-aware allocators that account for the non-additive nature of layer restoration.

Summary updated

Abstract

Low-bit LLM quantization is useful only if a model preserves service-relevant accuracy, executes through supported GPU kernels, and meets measured memory and latency limits. We study Qwen3-8B on NVIDIA B200 with vLLM, including INT2/INT3, GPTQ, NVFP4, and mixed-precision configurations. We verify native packed execution and precision on all 144 fused linear modules and evaluate Korean, English, numerical, and reasoning quality under preregistered gates. Packed INT2 and INT3 execute, but all seven tested INT2 grids degenerate on generation. Isolated module-restoration effects are strongly non-additive; Korean and reasoning sensitivity rankings disagree, and a promising INT4/INT8 development candidate fails independent final evaluation. A configuration with 16-bit activations, 98 NVFP4 weight-only modules, 46 INT8 modules, and an INT8 output head passes all gates on the untouched FINAL-81 set (5,168 items): 5.917 GiB of weight-loading memory (limit 5.98), Korean teacher-correct retention 0.904 (gate 0.90), and TPOT ratio 0.939 relative to BF16. It improves paired accuracy over NVFP4 W4A4 by +1.4 percentage points (Bonferroni-adjusted interval [+0.2, +2.6]), compared with +3.4 points on the selection set. This comparison changes both activation precision and INT8 protection; exploratory analyses suggest that the activation change contributes substantially, but the isolated allocation effect is not established. Final checkpoints were not retained, and bitwise rebuild reproducibility remains unverified.
BibTeX
@misc{han2026beyond,
  author = {Hyojung Han},
  title = {{Beyond Bitwidth: Deployment-Constrained Mixed-Precision Quantization for Multilingual LLM Inference on NVIDIA Blackwell}},
  year = {2026},
  eprint = {2611.21974},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.21974},
}
Figure