Quantization—the process of reducing the numerical precision of neural network weights and activations—is the primary tool for deploying massive models on edge devices. However, quantization error does not distribute itself uniformly across a network. Instead, specific architectural components act as "sensitivity bottlenecks" where even minor precision loss can lead to a total collapse in model performance.
The research direction across these 90 papers focuses on identifying these high-sensitivity regions and developing mixed-precision strategies to protect them. By keeping a small fraction of critical layers in higher precision while aggressively compressing the rest, researchers can achieve massive speedups without sacrificing accuracy.
The Boundary Effect: Why First and Last Layers Matter
The most consistent finding across the literature, from early binarization work like XNOR-Net by Mohammad Rastegari to modern LLM studies, is that the initial and final layers are disproportionately sensitive to bitwidth reduction.
The input layer is responsible for capturing raw signal integrity. As Yong Liu notes, quantizing raw pixels or audio samples too early destroys essential information before the network can extract meaningful features. Similarly, Zefan Li in High-Order Residual Quantization points out that initial layers often have very few input channels (e.g., 3 for RGB), meaning there is no "redundancy" to buffer the noise introduced by low-bit rounding.
The output layer serves as the final resolution for predictions. In classification, Shuchang Zhou (DoReFa-Net) demonstrated that high precision is required to maintain the subtle differences between class probabilities. In Large Language Models, Mingyang Wang found that the final layers are the "knowledge-safe" zones; quantizing them creates a bottleneck that prevents the model from correctly surfacing factual information, even if that knowledge is perfectly preserved in the internal layers.
Structural Fragility in Information Flow
Beyond the boundaries, certain internal structures are mathematically predisposed to failure under quantization.
1. The Down-Projection Bottleneck
In architectures like Mixture-of-Experts (MoE) or Transformers, data is often projected into a high-dimensional space and then "squeezed" back down. Zhanhao Xie discovered that these down-projection layers are significantly more sensitive than up-projection layers. Because they are tasked with compressing high-dimensional information back into a smaller hidden dimension, any error introduced by quantization is amplified during the dimensionality reduction.
2. Depth-wise Convolutions and Low Redundancy
Efficient models like MobileNet are ironically the hardest to quantize. Eunhyeok Park (PROFIT) identified that depth-wise convolutions suffer from "activation instability." Because these layers have fewer parameters than standard convolutions, they lack the representational redundancy needed to "average out" quantization noise, causing output activations to fluctuate wildly.
3. Residual Shortcuts
Residual connections are the highways of information flow. Yiren Zhao (LPGNAS) found that these shortcuts are "critical paths." If a shortcut is quantized too aggressively, the error bypasses the transformation layers that might have otherwise corrected the signal, leading to a compounding degradation across the entire network.
The Reversal Effect: How Normalization Amplifies Noise
Recent work on LLMs has uncovered a counter-intuitive "reversal effect" that explains why some inputs break quantized models while others don't. Ting-Yun Chang and Muru Zhang in Why Do Some Inputs Break Low-Bit LLM Quantization? found that inputs with smaller residual stream magnitudes are actually more vulnerable.
When the internal signal is small, RMSNorm (Root Mean Square Layer Normalization) applies a large scaling factor to normalize the activation. This scaling doesn't just boost the signal; it also amplifies the quantization noise in the weights. This noise is then pushed into the MLP gate projections—which Muru Zhang identifies as the most sensitive component in the Transformer block—causing the model to produce incoherent "gibberish" outputs.
Strategies for Sensitivity-Aware Compression
To combat these vulnerabilities, the field has moved toward automated sensitivity analysis and mixed-precision deployment:
- Loss-Aware Quantization: Instead of minimizing the error between full-precision and quantized weights, methods like ALQ by Zhongnan Qu directly minimize the error on the loss function. This naturally identifies which layers the model "cares" about most.
- Progressive Training: For complex models like Diffusion Models, Haotong Qin (BinaryDM) suggests an "evolvable" strategy, where the most sensitive layers are trained with high-capacity mimicking before the rest of the model is binarized.
- Bit-Gradient Analysis: Souvik Kundu (BMPQ) uses gradients at the bit-level to automatically determine the optimal bitwidth for each layer, ensuring that "expensive" bits are only spent where they provide the most accuracy gain.
While these strategies allow for high-performance compressed models, they create a "hardware floor." As Yaman Umuroglu points out, keeping the first and last layers in high precision can account for a disproportionate share of execution time, meaning the ultimate goal of the field remains finding ways to stabilize even these most sensitive layers at ultra-low bitwidths.