LittleBit: Ultra Low-Bit Quantization via Latent Factorization

Banseok Lee, Dongkyu Kim, Youngcheon You, Youngmin Kim

TL;DR

The paper introduces LITTLEBIT, a novel framework for extreme large language model (LLM) compression that pushes quantization down to an unprecedented 0.1 bits per weight (BPW). By combining low-rank latent matrix factorization, multi-scale compensation, and a dual-path residual mechanism, LITTLEBIT shrinks models like Llama2-13B to under 0.9 GB while unlocking up to an 11.6× inference speedup over FP16 precision.

What Problem Does This Paper Address?

Deploying large language models (LLMs) on consumer or edge devices is heavily bottlenecked by memory (VRAM) and computational costs. While standard quantization techniques can compress models to 4 bits with minimal loss, pushing past the 1-bit barrier (sub-1-bit regime) typically leads to catastrophic performance degradation.

At ultra-low bit-widths, the massive information loss causes severe training instability in Quantization-Aware Training (QAT) and massive accuracy drops in Post-Training Quantization (PTQ). The authors tackle this "quantization cliff," aiming to achieve extreme compression (down to 0.1 BPW) while maintaining model viability and reasoning capabilities.

Methods

LITTLEBIT completely redesigns the linear layers of a Transformer using a combination of matrix factorization, binarization, and error correction:

  1. Latent Matrix Factorization & Binarization: Instead of quantizing the high-precision weight matrix $W$ directly, LITTLEBIT factorizes it into two low-rank latent matrices ($W \approx UV^\top$) and subsequently binarizes these factors using a element-wise sign function ($\pm1$).
  2. Multi-Scale Compensation: To counteract the information loss from binarization, the framework introduces learnable FP16 scaling vectors across three distinct dimensions: rows ($h$), columns ($g$), and a novel latent dimension ($\ell$) that scales the columns of the binarized factors based on importance.
  3. Dual-SVID Initialization: Naively initializing such a constrained structure causes unstable training. The authors propose Dual Sign-Value-Independent Decomposition, which uses a truncated SVD on the original weights to cleanly disentangle and initialize the sign and magnitude components.
  4. Residual Compensation: Rather than relying on a single high-rank approximation, LITTLEBIT splits its fixed bit budget into two parallel, lower-rank pathways: a Primary path and a Residual path. The residual path specifically learns to correct the approximation errors made by the primary path.
  5. SmoothSign Gradient Proxy: For backpropagation through the non-differentiable sign function, the authors replace the standard Straight-Through Estimator (STE) with a smooth proxy gradient based on the derivative of $\tanh(100x)$, which provides superior stability.
Comparison of a standard Transformer layer and the LITTLEBIT architecture featuring parallel Primary and Residual pathways.

Datasets

The authors evaluated LITTLEBIT across several prominent open-source LLM families (Llama, Llama2, Llama3, OPT, Phi-4, and QwQ) spanning scales from 1.3B to 32B parameters.

  • Language Modeling (Perplexity): Evaluated on WikiText-2, C4, and PTB validation sets.
  • Zero-Shot Reasoning Benchmarks: Evaluated across 7 common-sense tasks: WinoGrande, OpenBookQA (OBQA), HellaSwag, BoolQ, ARC-Easy (ARC-e), ARC-Challenge (ARC-c), and PIQA.

Key Insights and Ideas

  • Factorization Beats Pruning at the Extreme: The authors observe that LLM weights possess a pronounced low-rank structure. At extreme compression ratios (e.g., retaining less than 25% of parameters), SVD-based factorization maintains model integrity far better than state-of-the-art pruning methods like Wanda.
  • The Power of the Latent Scale: Standard low-bit methods scale rows and columns. LITTLEBIT proves that adding a third scaling dimension—the latent dimension ($\ell$)—is crucial for capturing the multi-dimensional magnitude variations inherent in factorized representations.
  • Two Paths Are Better Than One: Splitting the parameter budget into a primary path and an explicit error-correcting residual path yields a much more faithful initial weight representation than a single, higher-rank path.

Key Results

  • Robustness in the Sub-0.5 BPW Regime: Where prior state-of-the-art sub-1-bit methods (like STBLLM) suffer severe performance collapse, LITTLEBIT remains stable. For example, on Llama2-13B, LITTLEBIT at an extreme 0.1 BPW achieves a perplexity of 15.09, outperforming STBLLM operating at a much higher 0.55 BPW (27.05).
Figure 1: Low-bit quantization perplexity on Llama2-13B (WikiText-2). LITTLEBIT surpasses the state-of-the-art sub-1-bit quantization technique. Below 0.5 BPW, where the leading prior method degrades sharply, ours remains robust down to 0.1 BPW.
  • Preserved Reasoning Capabilities: Zero-shot evaluations on Phi-4 (14B) and QwQ (32B) show that LITTLEBIT maintains viable reasoning performance down to 0.1 BPW, whereas baseline methods collapse completely below 0.5 BPW.
Figure 4: Zero-shot accuracy (%) on 7 common sense reasoning tasks. The comparison highlights the robustness of LITTLEBIT (solid lines) against STBLLM (dashed lines) in the sub-0.5 BPW regime. While STBLLM degrades sharply, LITTLEBIT preserves viable reasoning capabilities for both (a) Phi-4 and (b) QwQ models.
  • Massive Hardware Speedups: Using a custom-developed 1-bit GEMV CUDA kernel on an NVIDIA A100 GPU, LITTLEBIT achieved up to an 11.6× kernel-level speedup over an optimized FP16 baseline for a Llama2-70B MLP layer.
  • KV Cache & Memory Reduction: When applied to key/value projections, the factorization inherently compresses the KV cache by up to 21.3× for Llama2-7B at 0.1 BPW. It also compresses the total memory footprint of Llama2-70B from 138.04 GB down to under 2 GB.

Limitations and Future Work

  • The lm_head Bottleneck: LITTLEBIT focuses entirely on compressing the core Transformer blocks. It leaves the language model head (lm_head) in uncompressed FP16 format, which becomes a significant storage bottleneck for models with large vocabularies at ultra-low bit rates.
  • QAT Compute Costs: Quantization-Aware Training is computationally heavy. The authors note they faced resource constraints when attempting to scale QAT to models of 70B parameters and beyond, highlighting a need for more resource-efficient training or post-training adaptations.
  • Theoretical Gaps: While empirically successful, a deeper information-theoretic understanding of how these models retain capabilities when stripped of 99% of their original weight information remains an open question.

Summary updated

Abstract

The deployment of large language models (LLMs) is frequently hindered by prohibitive memory and computational requirements. While quantization mitigates these bottlenecks, maintaining model fidelity in the sub-1-bit regime remains a persistent challenge. In this paper, we introduce LittleBit, a novel framework for extreme LLM compression. We target quantization rates as low as $0.1$ bits per weight (BPW), achieving a memory reduction of approximately $31\times$, which effectively compresses Llama2-13B to under $0.9$ GB. We represent weights via low-rank latent matrix factorization and subsequently binarize the resulting factors. To counteract the information loss inherent to such drastic precision reduction, we integrate a multi-scale compensation mechanism that learns importance parameters across row, column, and latent dimensions. Two primary contributions enable effective training: Dual Sign-Value-Independent Decomposition (Dual-SVID) for quantization-aware training (QAT) initialization, and Residual Compensation to minimize approximation errors. Extensive experiments confirm the superiority of LittleBit in the sub-1-bit domain; for instance, our method at $0.1$ BPW surpasses the performance of leading techniques operating at $0.7$ BPW on Llama2-7B. We establish a new size-performance trade-off -- unlocking a potential $11.6\times$ inference speedup relative to FP16 -- and render powerful LLMs practical for resource-constrained environments. Our code is available at https://github.com/SamsungLabs/LittleBit.
BibTeX
@inproceedings{lee2025littlebit,
  author = {Banseok Lee and Dongkyu Kim and Youngcheon You and Youngmin Kim},
  title = {{LittleBit: Ultra Low-Bit Quantization via Latent Factorization}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2025},
  url = {https://openreview.net/forum?id=zJzu9evD5K},
}
Figure