Shrinking a large language model's weights from 16 bits to 4 bits barely hurts it. Going to 2 bits often breaks it outright: perplexity jumps by orders of magnitude, and multi-step reasoning collapses while fluent-sounding text survives. The most successful recent repairs, such as compensating each layer for error that arrives from earlier quantized layers, rest on an assumption nobody checks. They assume that small errors pass through the network roughly linearly. In other words, each layer is treated as its local linear approximation (its Jacobian, the matrix of first derivatives) evaluated at the full-precision activations. That is the assumption behind local stability analysis in dynamical systems, and that field has a known blind spot: once a perturbation is no longer small, the linearization says nothing.
We propose that the 2-bit cliff is this blind spot appearing inside a transformer. Below the cliff, quantization error moves through the layers as the per-layer Jacobians predict, so linear compensation works. At the cliff, a few specific layers and token positions receive error larger than the radius over which their linearization holds, and the network leaves the regime where any linear correction can help. The practical consequence is a different rule for spending bits. Instead of protecting the parameters that are most sensitive, we protect the places whose linear-validity radius is smallest. The goal is to keep every layer's incoming error inside the region where linear compensation still works.
Core Claim
Treat the stack of residual blocks as a discrete-time system indexed by depth. The state is the residual stream $x_l$, and quantization injects state-dependent noise at each step. Let $\delta_l$ be the deviation of the quantized model's hidden state from the full-precision one at layer $l$, and let $J_l$ be the Jacobian of block $l$ at the full-precision state. We measure a per-layer, per-token linearity gap
$$\gamma_l = \frac{\|\delta_{l+1} - (J_l \delta_l + e_l)\|}{\|J_l \delta_l + e_l\|},$$
where $e_l$ is the fresh error that layer $l$'s own quantized weights inject. We also measure a validity radius $r_l$, defined as the largest input-error norm for which $\gamma_l$ stays below a fixed tolerance $\tau$ (we use $\tau = 0.1$) on calibration data:
$$r_l = \sup\{\rho : \gamma_l \le \tau \text{ whenever } \|\delta_l\| \le \rho\}.$$
The claim has three falsifiable parts. First, at 3 bits $\|\delta_l\| < r_l$ almost everywhere, while at 2 bits the violations concentrate in a small set of layers and token positions. We expect the outlier-carrying "pivot" tokens noted by FlatQuant and the blocks holding CherryQ's high-impact parameters to dominate, but we treat this as a hypothesis, not an assumption. Second, the size of the perplexity blow-up is predicted better by these violations than by the summed sensitivity of the error. Third, if a fixed bit budget is spent to remove the violations, the result beats the same budget spent by sensitivity ranking.
Relation to Prior Work
Two papers motivate this work by documenting compounding error. Quantization Error Propagation (QEP) reformulates layer-wise post-training quantization to match the quantized layer's output, computed on corrupted upstream input, against the clean output. It then derives a closed-form correction scaled by a propagation strength $\alpha_l$ that defaults to 0.5. This rescues 2-bit QuIP on Llama-2-7B from perplexity 17,783 to 11.97. QEP's correction is a linear-regime method. Its own limitations section says it has no principle for setting $\alpha_l$. We do not propose a better $\alpha_l$. We propose a test of when any $\alpha_l$ can work.
Trajectory-based Quantization Sensitivity (TQS) borrows finite-time Lyapunov exponents to rank layers of weather and time-series forecasters. It measures how fast small injected Gaussian noise grows over autoregressive rollouts, then allocates precision with a knapsack solver. TQS is the closest existing work. It works along the time axis, not depth, and does not study language models. More importantly, it measures growth rate of small perturbations, which is a linear quantity. Our quantity is the breakdown of linearity at realistic error sizes. The two can disagree: a layer can amplify small errors modestly yet saturate abruptly under large ones, for example in a softmax or a normalization dominated by one outlier channel.
Three papers supply methods we reuse. Rotation methods (QuIP, QuaRot) spread energy evenly across coordinates before rounding, and they serve as our base quantizers. Non-Euclidean contraction theory shows that contraction measured in $\ell_\infty$-type norms gives bounds on how bounded disturbances spread (input-to-state stability). We use $\ell_\infty$ alongside $\ell_2$ when computing $\gamma_l$ and $r_l$, because outlier channels make the max-norm the natural measure.
One paper constrains the design. Learning Neural Contracting Dynamics shows that enforcing global contraction limits what a learned system can represent. We therefore do not try to make the transformer contractive. We only certify where its local linearization is trustworthy.
Research Program
The first stage is measurement. For Llama-2-7B, Llama-3-8B and Llama-2-70B, we quantize with QuIP and with GPTQ at 4, 3 and 2 bits, with and without QEP. We record $\delta_l$, $e_l$ and Jacobian-vector products $J_l\delta_l$ on 128 C4 calibration sequences. Jacobian-vector products cost about one extra forward pass per layer, so no explicit Jacobian is ever formed. We estimate $r_l$ by rescaling the true error direction and bisecting on its magnitude. Using the true direction rather than random noise matters, because quantization error is not isotropic. From these measurements we build maps of $\gamma_l$ and $r_l$ over layers and token positions, broken down by sublayer: attention softmax, normalization, and the MLP nonlinearity.
The second stage is prediction. We test whether a scalar summary, the fraction of (layer, token) pairs with $\|\delta_l\| > r_l$, predicts final WikiText-2 perplexity and GSM8K accuracy across bit-widths and quantizers. We compare it against two other predictors: TQS-style noise-growth scores computed along depth, and summed Hessian or impact scores in the style of CherryQ.
The third stage is intervention. Under a fixed average bit budget (2.0, 2.25 and 2.5 bits per weight), we allocate extra precision at block granularity. This keeps kernels uniform within a block and avoids the runtime-overhead problem of unstructured mixed precision. The allocation runs a greedy knapsack that, at each step, upgrades the block that most reduces radius violations. We then apply QEP's linear correction to the result. The prediction is that QEP's benefit grows sharply once violations are removed, because its linear model now holds.
As a secondary extension, we repeat the measurement along the decoding-time axis for KV-cache quantization. Evaluating Quantized LLMs reports that 4-bit KV caches collapse beyond 4K tokens. This extension tests whether the same radius test locates that failure.
Baselines
The naive reference point is uniform 2-bit QuIP and uniform 2-bit GPTQ with no propagation correction. It anchors how bad the cliff is under our exact setup. As external context only, CherryQ reports AWQ at 2 bits on LLaMA2-7B exceeding $10^5$ perplexity, and QEP reports 17,783 for standard 2-bit quantization of Llama-2-7B.
The strongest post-training competitor is QuIP plus QEP. QEP reports a perplexity of 11.97 for 2-bit QuIP with QEP on Llama-2-7B, listed in the file under WikiText-2. We will rerun it ourselves with the same calibration set and the same evaluation script. We will also give QEP a per-layer $\alpha_l$ grid search with calibration compute equal to the cost of our radius estimation. Beating it means lower WikiText-2 and C4 perplexity and higher GSM8K accuracy at an identical average bit-width, across at least two model sizes.
The strongest allocation competitor is a depth-wise TQS adaptation: Gaussian-noise growth scores feeding the same knapsack solver at the same block granularity and budget. This baseline isolates our key ingredient. It shares everything with our method except the scoring rule, so it compares sensitivity to small noise against linear validity at the actual error size.
A second allocation baseline is CherryQ-style impact scoring, restricted to block granularity for fairness. For orientation, CherryQ's own unstructured result is 9.55 perplexity at 2 bits on LLaMA2-7B. That number is reported in the file without a dataset label, and the same paper's 3-bit numbers are on C4.
Fine-tuning methods form a separate tier and are reported but not claimed as targets. PV-Tuning reports WikiText-2 perplexity 5.71 at about 2 bits on Llama-2-7B, versus 6.19 for QuIP# and 6.64 for AQLM. These use far more compute than post-training quantization. We will test whether radius-guided allocation also helps when used as the starting point for PV-Tuning.
The ablation that matters most keeps radius-guided allocation but drops QEP correction afterward. It tests whether the gain comes from restoring linear validity, which should make QEP more useful, or simply from better allocation.
Why not just…?
Why not just tune QEP's $\alpha_l$ per layer by grid search? This is the most direct threat, and the program includes it as a baseline with matched compute. Our hypothesis says tuning a linear gain cannot fix layers where the error has already left the linear regime. If per-layer tuning closes most of the gap to our method, the hypothesis is wrong for these models and we will report that. The measurement stage settles the question in advance: if violations of $r_l$ are rare at 2 bits, tuning $\alpha_l$ should suffice.
Why not just use TQS-style noise sensitivity along depth? TQS found that its dynamical rankings disagree with static curvature rankings, which supports looking at propagation at all. However, it injects small Gaussian noise, which measures linear growth. Our claim is that the cliff comes from nonlinear saturation at realistic error sizes. The matched-allocation comparison in the third stage decides between the two. If the two rankings agree and perform equally, the validity radius adds nothing and the idea collapses into a depth-wise TQS.
Why not just keep the high-impact "cherry" parameters in 16 bits? CherryQ works well at 3 bits. Its 2-bit result still lags rotation-plus-propagation methods, and unstructured retention needs special kernels. If the radius violations turn out to sit exactly on cherry-parameter blocks, our method reduces to a structured, cheaper version of CherryQ. That would still be a useful explanation of why CherryQ works, but a weaker methodological claim. The overlap between the two maps is reported directly.
Why not just fine-tune with quantization-aware training? PV-Tuning shows that plain straight-through gradient estimation is unstable at 1 to 2 bits, and CherryQ reports standard quantization-aware training at 3 bits reaching 9.25 C4 perplexity versus 7.39 for CherryQ. Fine-tuning is also a different compute regime. We test whether our allocation improves PV-Tuning's starting point rather than competing with it.
Decisive Checks
The first check comes before any method claim. On Llama-2-7B and Llama-3-8B, we will report whether radius violations rise sharply between 3 and 2 bits while linear noise-growth scores change smoothly. A smooth rise in violations would undercut the cliff explanation.
The second check is predictive. The rank correlation between violation fraction and final perplexity, across all quantizer, bit-width and model combinations, must exceed that of the depth-wise TQS score and the summed impact score.
The third check is the intervention at 2.0 to 2.5 average bits. Radius-guided allocation followed by QEP must beat TQS-guided allocation followed by QEP, and must beat QEP with tuned $\alpha_l$. The comparison covers WikiText-2 and C4 perplexity, the zero-shot suite (PIQA, ARC, HellaSwag, WinoGrande, LAMBADA) and GSM8K, which is the benchmark Evaluating Quantized LLMs identifies as most fragile. It must hold on at least two of three model sizes, with three calibration seeds.
Failing the third check while passing the first two would leave a diagnostic contribution, an account of the cliff, without a better quantizer. We would report it that way.
References
- FlatQuant: Flatness Matters for LLM Quantization
- Quantization Error Propagation: Revisiting Layer-Wise Post-Training Quantization
- Quantizing Time-Series Models As Dynamical Systems: Trajectory-Based Quantization Sensitivity Score
- QuIP: 2-Bit Quantization of Large Language Models With Guarantees
- QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs
- Non-Euclidean Contraction Theory for Robust Nonlinear Stability
- Learning Neural Contracting Dynamics: Extended Linearization and Global Guarantees
- Cherry on Top: Parameter Heterogeneity and Quantization in Large Language Models
- Evaluating Quantized Large Language Models