MLWQ: Efficient Small Language Model Deployment via Multi-Level Weight Quantization

Chun Hu, Junhui He, Shang-Yu Wu, Yuxin He, Chun Jason Xue, Qingan Li2025 · DOI

Introduction

While Large Language Models (LLMs) dominate the spotlight, Small Language Models (SLMs) like Llama-3.2-1B and Phi-2 are the practical workhorses for on-device AI. However, deploying these models on resource-constrained edge devices remains a challenge. Post-Training Quantization (PTQ) is the standard approach to reduce memory footprints by lowering weight precision (e.g., from 16-bit to 4-bit).

Existing PTQ methods often suffer from significant accuracy degradation when pushing below 4 bits because they treat all layers uniformly or use simplistic binary classifications for weight importance. The MLWQ (Multi-Level Weight Quantization) framework addresses these gaps by jointly optimizing bit-widths across layers and refining weight importance within each layer.

Methodology

MLWQ introduces a three-stage optimization pipeline to achieve high compression without sacrificing intelligence.

Bit-width Preallocation based on Layer Loss (BPLL)

Not all layers respond to quantization equally. Some are highly sensitive to precision loss, while others are more robust. MLWQ calculates the potential loss for each layer at varying bit-widths and solves a constrained optimization problem to allocate a global "bit budget."

Loss sensitivity across different layer types at 2-bit and 4-bit precision.

As shown in the sensitivity analysis, certain components like FC1 exhibit extreme loss at 2-bit precision compared to others like the 'O' (output) projection. By identifying these bottlenecks, BPLL ensures sensitive layers receive higher bit-widths (e.g., 4-bit) while robust layers are compressed further (e.g., 2-bit).

Mixed-precision Quantization based on Salience Awareness (MQSA)

Within individual layers, weight importance exists on a spectrum rather than a binary "salient vs. non-salient" toggle. MLWQ partitions weights into three categories—Salient, Ordinary, and Non-salient—based on Hessian-based importance metrics.

Salience distribution within the output projection layer, highlighting non-uniform weight importance.

By assigning different bit-widths (e.g., 4, 3, and 2 bits) to these groups, MQSA provides a "middle ground" for ordinary weights, smoothing the precision gap and stabilizing model performance. This granular approach captures the nuance of the weight landscape better than traditional methods.

Tweaking Quantization Parameters (TQP)

The final stage involves fine-tuning the clipping thresholds for each of the three weight groups. This granular adjustment minimizes local quantization error by ensuring the quantizer's range is perfectly tailored to the distribution of each weight category.

Key Results

MLWQ consistently outperforms state-of-the-art baselines like OPTQ and AWQ across various SLM families (OPT, Llama-3.2, Phi, SmolLM2):

  • Perplexity (PPL): On OPT-125M at ~3-bit quantization, MLWQ achieved a perplexity of 28.98, a massive improvement over OPTQ (53.37) and SliM-LLM (43.41).
  • Reasoning Accuracy: For Llama-3.2-1B, MLWQ reached a zero-shot average accuracy of 42.60%, outperforming OWQ (42.46%) while maintaining a lower overall bit-rate of 2.99 bits.
  • Inference Efficiency: GPU deployment tests indicate that MLWQ maintains inference speeds comparable to existing frameworks while providing significantly higher model quality.

Conclusion and Future Work

MLWQ demonstrates that the key to aggressive SLM compression lies in recognizing both inter-layer and intra-layer sensitivity. By treating weight importance as a spectrum and optimizing bit-allocation globally, MLWQ enables high-fidelity 3-bit deployment on edge devices.

Future research will focus on addressing activation overhead and optimizing specialized CUDA kernels to further reduce the decoding overhead associated with mixed-precision branching, ensuring theoretical memory savings translate into maximum hardware speedups.

Summary updated

BibTeX
@misc{hu2025mlwq,
  author = {Chun Hu and Junhui He and Shang-Yu Wu and Yuxin He and Chun Jason Xue and Qingan Li},
  title = {{MLWQ: Efficient Small Language Model Deployment via Multi-Level Weight Quantization}},
  year = {2025},
  doi = {10.18653/v1/2025.emnlp-main.408},
}
Figure