Quantization is the process of reducing the numerical precision of a neural network's weights and activations to save memory and speed up inference. While moving from 32-bit floating point to 8-bit integers is often straightforward, pushing models into the "ultra-low-bit" regime—4-bit, 2-bit, or even 1.58-bit—usually results in a catastrophic collapse of reasoning and accuracy.
The emerging consensus across recent research is that standard training objectives, like next-token prediction or hard-label classification, provide a signal too sparse to repair the damage caused by aggressive bit-width reduction. Instead, researchers are increasingly using Knowledge Distillation (KD) as a corrective buffer. By treating a high-precision model as a "teacher" and the quantized model as a "student," the student can inherit the nuanced logic of the original, even when its own representational capacity is severely constrained.
The Information Density of Soft Targets
In a standard training setup, a model learns from "hard labels"—binary indicators of the correct class. However, as Jung Hyun Lee and colleagues argue in Unified Progressive Quantization, hard labels lack the informational density required to fix the logic errors introduced by 2-bit quantization.
When a student model is quantized, it loses the ability to represent subtle differences in probability. Knowledge distillation solves this by forcing the student to match the teacher's full probability distribution (the "soft labels"). This distribution contains "dark knowledge"—information about which incorrect classes are more likely than others—which acts as a dense supervisory signal. Boris van Breugel and the team behind FPTQuant demonstrate that matching this full distribution allows 4-bit LLMs to maintain near-zero accuracy loss by preserving the mathematical symmetries of the original architecture.
Aligning Internal Representations
The synergy between quantization and distillation extends beyond the final output. To prevent performance collapse in complex tasks like speech recognition or graph analysis, the student must mimic the teacher’s internal "thought process."
For instance, Abdelrahman Mohamed found that for low-bit speech models, mimicking the teacher's internal attention patterns and hidden states is more effective than relying on final labels. Similarly, Fayao Liu introduced Graph Contrastive Representation Distillation, which uses contrastive learning to ensure a lightweight, quantized Graph Neural Network (GNN) preserves the "global topology" of a larger teacher. By aligning these high-dimensional embeddings, the student retains critical structural information that would otherwise be discarded during quantization.
Bridging the Capacity Gap
A significant challenge in this area is the "capacity gap": a 1-bit student may be physically incapable of mimicking the complex, high-precision outputs of a 32-bit teacher. If the teacher is too "smart," its signals become noise to a low-capacity student.
To solve this, researchers use two primary strategies:
- Progressive Distillation: Instead of jumping from 32-bit to 2-bit, the model is distilled through intermediate stages. Haichao Yu (in Any-Precision DNNs) and Bohan Zhuang (in Switchable Precision Neural Networks) utilize a hierarchy where higher-precision configurations of the same model act as teachers for lower-precision versions. This ensures the student always has a "relatable" target.
- Teacher Adaptation: In Quantization-aware Knowledge Distillation, Jangho Kim proposes updating the teacher alongside the student. This forces the teacher to simplify its own distribution, making its knowledge more "digestible" for a student with limited representational power.
Synergy in Training Loops
The most effective implementations, such as DQ-BART by Zheng Li and colleagues, integrate distillation directly into the Quantization-Aware Training (QAT) loop. In this framework, the model uses quantized weights for the forward pass—simulating the errors of low-precision hardware—while using the teacher's guidance to adjust full-precision weights during the gradient update.
This creates a corrective cycle: the distillation process guides the student to find "quantization-robust" features, while the quantization constraints force the student to focus only on the most essential information provided by the teacher. As demonstrated by Shaohan Huang in BitNet Distillation, this approach can scale even to 1.58-bit LLMs, achieving massive reductions in memory and latency with performance nearly identical to full-precision baselines.