GateTrim: Adaptive-Rank Quantization for Large Language Models

Hongwei Ren, Jincheng Ren, Yuchuan Zhou, Jingchun LingarXiv 2026 · arXiv

TL;DR

GateTrim is an adaptive-rank post-training quantization (PTQ) method that learns exactly how much "extra" capacity each weight matrix in a Large Language Model (LLM) needs to recover from quantization errors. By using learned gates and a shared threshold, it achieves a surprising "less is more" result: pruning more low-rank components (higher threshold) leads to both smaller model sizes and significantly better language modeling performance.

What Problem Does This Paper Address?

Quantizing LLMs—reducing the precision of weights and activations to 4-bit or 8-bit—is essential for deployment but inevitably introduces errors. A popular remedy is the low-rank-residual representation, where a weight matrix is represented as a dense quantized "residual" plus a low-rank branch that compensates for the quantization noise.

The challenge lies in capacity allocation. Most existing methods assign a fixed rank (e.g., 128) to every single layer in the model. However, not all layers are created equal; some might need significant help to recover their original function, while others might be perfectly fine with very little extra capacity. Using a fixed rank is inefficient, wasting memory on layers that don't need it and potentially under-serving those that do.

Methods

GateTrim introduces a three-stage calibration pipeline to learn matrix-specific ranks:

  1. Stage 1 (Factor Fit): The authors initialize a low-rank-residual decomposition and optimize the factors ($U, V$) and the residual ($R$) to minimize the reconstruction error of the original FP16 layer outputs.
  2. Stage 2 (Gate Calibration): They introduce a "gate" vector $z$ for each matrix. While keeping the weights fixed, they optimize these gates to learn the relative importance of each latent direction in the low-rank branch.
  3. Stage 3 (Compact Reconstruction): A shared threshold $\tau$ is applied to the gates. Any component with a gate value below $\tau$ is "trimmed" away. Finally, the remaining parameters are re-optimized (reconstructed) to compensate for the removed components.
The three-stage calibration process of GateTrim

Crucially, the method uses Z-score standardization on the low-rank factors before quantization, ensuring that the learned gates act on a normalized space.

Datasets

The authors evaluated GateTrim on the WikiText-2 dataset.

  • Calibration: They used a remarkably small "fitting pool" of just eight 512-token segments (4,096 tokens total) and a separate development pool of the same size.
  • Evaluation: Standard perplexity (PPL) testing on the WikiText-2 test set.
  • Models: The study covers a wide range of modern LLMs, including Qwen3 (4B, 8B, 14B), Llama-3.2-3B-Instruct, and Llama-3.1-8B-Instruct.

Key Insights and Ideas

  • The "Less is More" Paradox: The most striking finding is that increasing the pruning threshold from $\tau=0.3$ to $\tau=0.7$ actually improved performance. In all five models tested, the higher threshold resulted in a 46%–68% reduction in mean rank while simultaneously lowering WikiText-2 perplexity by up to 5.17%. This suggests that over-allocating rank can introduce noise or lead to overfitting during the calibration phase.
  • Heterogeneous Allocation is Key: The paper shows that a single threshold leads to very different ranks across the model. Some layers might retain a rank of 128, while others are trimmed down to 1. This confirms that fixed-rank methods are fundamentally sub-optimal.
  • Reconstruction Recovers Quality: The "Post" reconstruction stage (Stage 3) is vital. It allows the model to "heal" after the discrete step of rank selection, significantly lowering perplexity without increasing the storage footprint.

Key Results

The authors evaluated GateTrim in a W4A8 configuration (4-bit weights, 8-bit activations).

  • Perplexity Gains: On Qwen3-8B, the perplexity improved from 11.71 at $\tau=0.3$ to 11.24 at $\tau=0.7$.
  • Efficiency: Moving to the higher threshold reduced the measured target-export size by 1.50%–4.25%.
  • State-of-the-Art Comparison: In a cross-study comparison with TwinQuant (a recent high-performance quantization method), GateTrim achieved lower perplexity values across the Qwen3 family, despite using significantly less calibration data (4k tokens vs TwinQuant's 262k tokens).
Comparison of GateTrim results across different models and thresholds

Limitations and Future Work

  • Scope of Evaluation: The study is currently limited to W4A8 quantization and the WikiText-2 perplexity benchmark. The authors acknowledge that performance on downstream tasks (like MMLU or GSM8K) and instruction-following remains to be verified.
  • Simulated vs. Native Execution: The current implementation uses "floating-point QDQ" (simulated quantization). To realize actual speed and memory benefits on hardware, native low-bit kernels that support these adaptive-rank structures need to be developed.
  • Calibration Mismatch: Each matrix is calibrated against the original model's activations. In a real quantized model, errors accumulate across layers. Future work could explore "block-wise" or "end-to-end" calibration to account for this error propagation.

Summary updated

Abstract

We introduce GateTrim, an adaptive-rank post-training quantization method for large language models that learns matrix-specific ranks through gate optimization and threshold-based rank selection. GateTrim standardizes the low-rank factors of a quantized low-rank--residual representation, learns a separate gate vector for each target weight matrix, and uses a shared threshold to automatically determine its retained rank. The resulting ranks can differ across matrices, and the compact quantized representations are subsequently reconstructed. We evaluate GateTrim under W4A8 on Qwen3-4B/8B/14B, Llama-3.2-3B-Instruct, and Llama-3.1-8B-Instruct. Across all five models, we observe a counterintuitive finding: increasing the threshold from $τ=0.3$ to $τ=0.7$ reduces the mean retained rank by 46.60%--67.89% while simultaneously lowering WikiText-2 perplexity by 0.59%--5.17%. Threshold 0.7 achieves the lowest mean retained rank and final perplexity among the three tested thresholds, 0.3, 0.5, and 0.7, and reduces measured target-export size by 1.50%--4.25% relative to 0.3. GateTrim uses an $8\times512$-token fitting pool and a separate development pool of the same size. Its Qwen3-4B/8B/14B perplexities at $τ=0.7$ are numerically below TwinQuant's published W4A8 values, reported with a $128\times2048$-token calibration pool, in a cross-study comparison. On Qwen3-8B, the threshold-0.7 configuration already has the lowest perplexity before reconstruction under both tested initialization seeds; reconstruction improves it further without increasing retained rank or export size.
BibTeX
@misc{ren2026gatetrim,
  author = {Hongwei Ren and Jincheng Ren and Yuchuan Zhou and Jingchun Ling},
  title = {{GateTrim: Adaptive-Rank Quantization for Large Language Models}},
  year = {2026},
  eprint = {2611.00654},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.00654},
}
Figure