When a large language model is compressed by storing its numbers with fewer bits, it usually keeps its fluent grammar but loses multi-step reasoning. We propose a mechanistic explanation for this pattern and a test that could falsify it. To track nested structure such as matched brackets, a Transformer must make some attention heads almost "hard": each puts nearly all its weight on one earlier token. The margin by which that token must beat its competitors grows with the length of the context. Rounding noise from low-bit storage eats into that margin. Our prediction is that quantization does not degrade a model uniformly. Instead it caps the nesting depth and dependency length the model can track. A model that behaves like a stack machine at full precision ends up behaving like a bounded-depth, effectively finite-state approximation, and the cap moves predictably with bit-width.
This matters for two reasons. If the prediction holds, the field gets a controlled, measurable quantity, "effective depth at $b$ bits." That quantity can replace entangled benchmarks like competition math as the probe for the quantization cliff. It also says where scarce extra bits should go: to the query and key computations of a few depth-tracking heads, not to whichever weights look large on generic text. If the prediction fails, and accuracy instead drops uniformly across depths or only tracks sequence length, we learn that the "syntax survives, reasoning dies" story needs a different mechanism.
Core Claim
Quantization studies report a consistent dissociation. Models squeezed to about 2 bits keep their syntax and instruction format but lose logic, a pattern called the "Bucket Effect" in Quantization Meets Reasoning. An empirical study of LLaMA3 quantization and Evaluating Quantized Large Language Models report the same pattern. Quantization Hurts Reasoning? adds that harder benchmarks degrade up to 4x more than easy ones. All of these papers measure on natural-language tasks, where arithmetic, world knowledge and structure are mixed together, so none can say which computation breaks.
Formal-language work supplies the candidate mechanism. How Can Self-Attention Networks Recognize Dyck-n Languages? shows a two-layer self-attention network that recognizes Dyck languages. Dyck languages are strings of correctly matched brackets of several types, and recognizing them requires a stack. The network does this by making its second-layer attention "virtually hard" on the most recent unmatched opening bracket. That paper never touches numerical precision.
The link between the two is a simple softmax fact. Suppose a head must place probability at least $p$ on one target among $n$ visible positions, and the target's attention score exceeds the others by a margin $\Delta$. Then
$$\Delta \ge \log\frac{(n-1)\,p}{1-p}.$$
This follows directly from writing the softmax weight of the target as $e^{\Delta}/(e^{\Delta}+n-1)$. Quantizing the query weights, key weights or cached keys adds noise of scale $\sigma_b$ to the scores, and $\sigma_b$ roughly doubles per bit removed. The usable margin is therefore roughly $\Delta_{\text{FP}} - c\,\sigma_b$, where $\Delta_{\text{FP}}$ is the full-precision margin and $c$ is a constant to be fit. Requiring this usable margin to satisfy the inequality above gives the testable relation
$$\log n^*(b) \approx \Delta_{\text{FP}} - c\,\sigma_b,$$
where $n^*(b)$ is the longest context, or deepest nesting, at which the stack-tracking head still works at $b$ bits.
Tasks a model solves with local, finite-state heuristics should be nearly flat in $b$, because they do not need hard attention over long ranges. Deep nested and cross-serial dependencies should show a "knee": a depth beyond which accuracy collapses, and which moves toward shallower depth as bits drop.
The new contribution is not "quantization hurts hard tasks." It is three specific things:
- A quantitative law connecting bit-width to an effective position on the expressivity hierarchy.
- Per-example prediction of failures from measured attention margins.
- A bit-allocation rule derived from that law.
Research Program
Build the probes. We assemble a graded suite from three existing synthetic benchmarks:
- Dyck-1 through Dyck-4 with the training lengths and length-generalization splits of Ebrahimi et al..
- The minimal-pair grammars of Targeted Syntactic Evaluation on the Chomsky Hierarchy: regular adjective chains, counting, nested (last-in-first-out) dependencies, and cross-serial (first-in-first-out) dependencies.
- The regular and deterministic context-free tasks of Neural Networks and the Chomsky Hierarchy.
Every item is stratified by length and nesting depth independently. This lets us separate "long" from "deep," which is the control a skeptic will demand.
Choose the models. We use three tiers:
- Small Transformers trained from scratch with and without the start symbol of Ebrahimi et al. Here the stack-tracking heads can be located exactly.
- Pythia-160M with and without k-Shuffle Dyck pre-pretraining, following Between Circuits and Chomsky. That paper found that attention heads formed on formal data are reused as a syntactic backbone. Our question is whether that backbone is a single point of failure under quantization.
- Off-the-shelf Llama-family models of the kind used in the quantization papers. These are probed with in-context bracket completion and the minimal-pair grammars.
Apply the quantizers. On every model we run four weight quantizers at 8, 4, 3 and 2 bits:
- simple round-to-nearest;
- GPTQ;
- QuIP, which randomly rotates weights to spread out large outlier values (QuIP);
- SliM-LLM, which gives more bits to groups of weights judged important (SliM-LLM).
We also apply key-value cache quantization separately. Cache quantization perturbs the keys directly, so it is the cleanest test of the margin mechanism.
Analyze the results. For each condition we measure two things. The first is the effective depth $d^*(b)$: the deepest stratum at which accuracy stays within 90% of full precision. The second is the score margin $\Delta$ of the identified stack-tracking heads on each example. We then fit the relation above and test whether low measured margin under quantization predicts which individual examples fail.
Build the intervention. The intervention follows from the law. We give extra bits only to the query and key projections, and the cached keys, of heads whose full-precision margin is close to the required threshold on long formal inputs. Everything else stays at the base bit-width, using SliM-LLM's group-aligned layout so the allocation remains hardware-friendly.
Baselines
The naive reference point is uniform round-to-nearest and uniform GPTQ at each bit-width, calibrated on WikiText2. These show the raw cliff. QuIP reports that at 2 bits GPTQ (called OPTQ there) reaches WikiText2 perplexity 123.91 on Llama 2 70B, against 6.33 for QuIP. That gap tells us uniform GPTQ will be a weak floor at 2 bits.
The strongest existing methods are QuIP and SliM-LLM. QuIP removes the outliers that most plausibly destroy large attention scores. SliM-LLM already allocates bits by importance and reports 48% lower perplexity than GPTQ for 2-bit LLaMA-7B. Neither paper reports any result on bracket-matching or Chomsky-hierarchy tasks; we found no reported result on Lacuna for quantized models on Dyck-n or the targeted minimal-pair benchmark. The full-precision anchor on Dyck is the Ebrahimi et al. result of 58.82% at out-of-distribution lengths 102–126 for Dyck-2. We rerun all methods on the same models with the same calibration set sizes and the same average bit budget, counting every extra bit our allocation spends.
The key ablation is "protect all attention query and key weights" at the same extra-bit budget. This isolates whether head selection by margin matters, or merely protecting attention. A second ablation picks heads at random.
What counts as winning. Our allocation beats SliM-LLM and QuIP if it raises $d^*$ by at least two nesting levels on Dyck and on the nested and cross-serial grammars at equal average bits, with WikiText2 perplexity no worse than 2% relative. It must also beat the all-attention ablation; otherwise the head-selection claim is dropped.
Why not just…?
Why not just conclude it is task difficulty or sequence length? That is the leading rival explanation. Quantization Hurts Reasoning? shows degradation scaling with difficulty. Our stratification by depth at fixed length, and by length at fixed depth, is the direct test. The margin theory predicts the knee follows the number of competing positions the head must suppress, not raw token count. If accuracy depends only on length and is flat in depth, our mechanism is wrong.
Why not just protect all attention weights at higher precision? This is a real threat. It is cheap, and it might capture most of the benefit. That is exactly why it is our main ablation. If it matches margin-selected heads at equal budget, we report the simpler rule and keep only the scientific claim about effective depth.
Why not just use QuIP's rotation, which already removes outliers? Rotation spreads large values across coordinates. That helps weight error on average, but it may also flatten the specific large score gaps that hard attention needs. Whether it preserves $\Delta$ is unknown. Running QuIP with and without our allocation on top settles whether its gains already include depth.
Why not just calibrate the quantizer on formal or reasoning data? Quantization Hurts Reasoning? reports that calibrating on math data rather than WikiText2 gains nearly 10%, and Between Circuits and Chomsky suggests hierarchical formal data shapes the right circuits. We therefore include GPTQ and SliM-LLM calibrated on k-Shuffle Dyck as baselines and as combinations with our method. Calibration can only reshape rounding within a fixed bit budget per layer. Our prediction is that it shifts the knee less than reallocating bits to near-threshold heads does. If formal calibration alone matches us, that becomes the finding.
Proposed Experiments and Decisive Checks
The first decisive check is the shape of the curve. In from-scratch Dyck models, we test whether $\log n^*$ falls roughly linearly in the measured score noise across 8 to 2 bits and across weight versus key-value cache quantization. We also test whether regular tasks stay near flat over the same range. The second half of this check uses the Delétang et al. regular tasks on which full-precision Transformers succeed within training lengths.
The second check is per-example prediction. Measured head margin under quantization should predict failure on held-out items. We compare it against a baseline predictor that uses only length and depth.
The third check transfers to pretrained models. On Llama-family models, $d^*(b)$ measured on the minimal-pair grammars should rank quantizers in the same order as their reasoning losses. A direct check is to compare against the reported MATH-500 drop, from 37.4% to 31.4%, for 4-bit Llama-3.2-3B in Quantization Meets Reasoning. A matching ranking would show that the formal probe captures what natural-language benchmarks entangle.
The fourth check tests formal pre-pretraining. We compare Pythia-160M with and without k-Shuffle Dyck pre-pretraining under 3-bit and 2-bit quantization on the same probes and on BLiMP. This tells us whether concentrating syntax into reused heads makes the model more fragile or more robust.
The fifth check is the allocation head-to-head against SliM-LLM, QuIP, formal-data calibration, and the all-attention ablation, at matched bits.
The proposal stands or falls on the first check. If no depth-dependent knee appears that moves with precision, the mechanism is wrong regardless of how the allocation performs.
References
- Quantization Meets Reasoning: Exploring LLM Low-Bit Quantization Degradation for Mathematical Reasoning
- An empirical study of LLaMA3 quantization: from LLMs to MLLMs
- Evaluating Quantized Large Language Models
- Quantization Hurts Reasoning? An Empirical Study on Quantized Reasoning Models
- How Can Self-Attention Networks Recognize Dyck-n Languages?
- Targeted Syntactic Evaluation on the Chomsky Hierarchy
- Neural Networks and the Chomsky Hierarchy
- Between Circuits and Chomsky: Pre-pretraining on Formal Languages Imparts Linguistic Biases
- QuIP: 2-Bit Quantization of Large Language Models With Guarantees
- SliM-LLM: Salience-Driven Mixed-Precision Quantization for Large Language Models