Is Quantization Damage Local in Depth? Measuring and Exploiting Cross-Layer Interactions for Mixed-Precision Allocation

When a network is compressed by storing its weights in fewer bits, some layers can tolerate 2 or 3 bits while others fall apart below 8. Mixed-precision methods try to give each layer the right number of bits under a total memory or latency budget. Almost all of them score each layer's sensitivity separately and add the scores. That additivity assumption is known to be wrong: the damage from quantizing two layers together is not the sum of the damage from quantizing each alone. Without additivity, choosing bits for $L$ layers from $B$ options means searching $B^L$ configurations. Explainable-AI research has faced the same exponential wall when splitting a prediction's credit among interacting input features. The escape it found was to assume that interactions have exploitable structure, most usefully that only nearby features interact, and then to check that assumption.

This proposal applies that move to network depth. We treat each layer's bit choice as a move in a cooperative game whose payoff is the model's output degradation. Using a modest number of measured quantized-model evaluations, we estimate how interaction strength decays with the distance between layers and whether a few "hub" layers break that locality. If the damage is mostly local in depth plus a few hubs, the allocation problem becomes a short-memory chain. A chain like this can be optimized exactly by dynamic programming under the budget constraint. If the damage is not local, we will have measured that fact directly, which the field currently lacks, and we will have shown which cheaper structural assumption does hold.

Core Claim

The claim is that quantization degradation, as a function of the per-layer bit vector $b = (b_1, \dots, b_L)$, is well approximated by three parts. The first is a cheap curvature prediction. The second is a correction for each layer on its own. The third is pairwise interactions restricted to layers within a small depth window $k$, plus pairwise terms touching a small set $H$ of hub layers:

$$ D(b) \approx F(b) + \sum_{i=1}^{L} u_i(b_i) + \sum_{0<|i-j|\le k} p_{ij}(b_i, b_j) + \sum_{h \in H}\sum_{j \ne h} q_{hj}(b_h, b_j). $$

Each symbol is defined as follows:

  • $D(b)$ is the measured KL divergence between the full-precision and quantized model's output distributions on calibration data.
  • $F(b)$ is the quadratic prediction from a Kronecker-factored Fisher estimate, cross-layer terms included.
  • $u_i$ are per-layer corrections.
  • $p_{ij}$ are local pairwise corrections.
  • $q_{hj}$ are long-range corrections through hubs.

What would be new has three parts. The first is an axiomatic measurement of how much quantization interaction lives at each depth distance and each interaction order. The second is the use of the curvature prediction as a control variate, so that the few measured evaluations only need to learn the residual. That residual is where quadratic approximations are known to fail: sub-4-bit perturbations and outlier-heavy layers. The third is an exact budgeted allocator whose cost scales as $L \cdot B^{k+|H|}$ times the budget resolution rather than $B^L$.

Where this sits in prior work

The motivation comes from three papers that found the gap. Augmenting Hessians with Inter-Layer Dependencies adds measured joint degradation of layer pairs to the diagonal Hessian and runs a bisection search. It shows that pairwise terms buy real latency, but it stops at order two and does not ask which pairs matter. Scalable Kronecker-Fisher Approximation shows non-diagonal cross-layer coupling up to 7B parameters, with Value projections acting as coupling hubs. This is exactly the evidence that pure locality may fail, and it is why the model above has a hub term. The same paper reports that the quadratic approximation underestimates Output-projection sensitivity because of outlier activations. That negative result constrains us: the curvature term cannot be trusted alone, so it is used only as a control variate. Q-Strata observes that quantizing one block changes another's sensitivity. It handles this with greedy descent on a model-level divergence, but it never characterizes the coupling itself.

Four papers supply the method. L-Shapley and C-Shapley restricts coalitions to graph neighborhoods under a Markov assumption for text and images. We apply the same restriction along network depth, where no one has tested it. Tractable Shapley Values and Interactions via Tensor Networks computes Shapley interaction indices of all orders from a low-rank multilinear surrogate, but only on tabular inputs. We use its tensor-train surrogate as the unrestricted, higher-order alternative against which locality is tested. ControlSHAP cancels Shapley sampling noise with a Taylor-expansion proxy and reports 80–95% variance reduction. We reuse that idea with the Fisher quadratic as the proxy.

Two papers are the closest game-theoretic compression work. MODEL SHAPLEY derives parameter Shapley values from a second-order expansion in one backward pass. It discards cross-block interactions and reports failure at INT4 because the scores are heavy-tailed. Shapley Pruning values channels by sampled coalitions for pruning. Our proposal differs from both in three ways. Its players are layers with multi-level bit choices. It measures degradation rather than extrapolating a Taylor series into the large-perturbation regime where that paper says the series breaks. And its output is an allocation with an exact optimizer, not an importance ranking.

Research Program

Players are the individual linear weight matrices. That is 7 per block in a Llama-style transformer, about 224 for a 32-block 7B model, and the convolutional layers for ImageNet CNNs. Bit choices are {2, 3, 4, 8}. We use group-wise round-to-nearest weight quantization with a fixed quantizer, so that differences between allocators come only from the bit assignment.

The first stage is the curvature prior. We compute $F(b)$ from Kronecker-factored Fisher blocks with cross-layer terms as in the Kronecker-Fisher paper. Hubs $H$ are chosen as the layers with the largest off-diagonal Fisher mass, at most three to four of them.

The second stage is the measurement design. We draw configurations from a stratified design: single-layer drops, pairs at controlled depth distances, and random mixed configurations at several average bit-widths. For each we measure $D(b)$ on 128 calibration sequences. The total budget is fixed at a few thousand evaluations, reported in GPU-hours.

The third stage is interaction analysis. For each drop level, for example 4→2, we define the binary game "which layers are dropped." From it we compute pairwise and third-order Shapley interaction indices of the residual $D - F$, using the tensor-train surrogate. We report interaction mass as a function of depth distance, interaction order, layer role (query/key/value/output/MLP), and drop level. This is the scientific product, and it is informative even if the allocator does not win.

The fourth stage fits the surrogate. The residual correction terms are fit by regularized least squares on the measured configurations.

The fifth stage is allocation. We run exact dynamic programming over layers in depth order. The state is the last $k$ bit choices plus cumulative bits, and the outer loop enumerates hub bit choices. For comparison, we also optimize the unrestricted tensor-train surrogate with the same budget constraint by beam search.

Baselines

The naive reference is uniform quantization at the matched average bit-width, for example uniform 3-bit or 4-bit. Any mixed-precision method must clear it, and it reveals how much allocation matters at all at each budget.

The standard independent-sensitivity method is HAWQ-V3-style allocation: per-layer Hessian trace sensitivity solved by integer linear programming, per HAWQ-V3. It is the fastest credible competitor and tests whether interactions matter.

The strongest existing interaction-aware method on CNNs and BERT is the Augmented Hessian search. It reports a 25.48% latency reduction on ResNet-50 at 99.99% of baseline accuracy, 21.69% on MobileNetV2 within 0.25% of baseline accuracy, and 33.28% on BERT/SQuAD at 99.99% of baseline accuracy. We rerun it with the same quantizer, the same 4,096-example calibration set it used, and a matched count of model evaluations. We beat it if, at equal average bits, our allocation gives higher accuracy with non-overlapping seed intervals, or matches accuracy at fewer bits.

On dense LLMs the strongest cheap competitor is NSDS. It is data-free and combines kurtosis with SVD structure. It reports gains of 2.26% on ARC-Challenge and 3.07% on PIQA over its zero-shot baseline for Qwen2.5-7B at a 3-bit budget, and stability down to about 2.6 bits. No reported absolute WikiText-2 perplexity for dense-LLM mixed-precision baselines was found on Lacuna, so we will run NSDS ourselves with its published settings.

A greedy search on true measured divergence, in the style of Q-Strata's outer loop, is the expensive upper reference. We run it under the same evaluation budget and also let it run until convergence. Beating it at equal evaluations counts as a win; matching it at convergence counts as a success for a much cheaper method.

Three ablations isolate the ingredients:

  • The Fisher prior alone ($D \approx F$), which tests whether measurements add anything.
  • The surrogate without the control variate, fit to $D$ directly, which tests the ControlSHAP ingredient.
  • An unrestricted sparse all-pairs surrogate fit with lasso on the same evaluations, which tests whether locality is doing the work or merely sparsity.

Why not just…?

Why not just use the Fisher or Hessian cross-terms directly, with zero measurements? This is the cheapest interaction-aware option. The Kronecker-Fisher paper itself reports that quadratic approximations underestimate Output-projection sensitivity under outlier activations. MODEL SHAPLEY reports that Taylor-based scores break at INT4. Both results suggest the failure concentrates at exactly the bit-widths where mixed precision matters most. The "Fisher alone" ablation settles this directly. If its held-out prediction of $D$ at 2–3 bits matches the full surrogate, the measurement stage is unnecessary, and we would report that.

Why not just fit a sparse regression over all layer pairs and skip the locality story? This is a real threat. With enough evaluations, lasso may find the same pairs without any structural assumption. We run it at identical evaluation budgets. If it matches both prediction and allocation quality, the finding becomes "quantization interactions are sparse but not local." The allocator would then switch to hub-plus-sparse integer programming. The interaction measurements remain the contribution either way.

Why not just run greedy search on the true divergence, as Q-Strata does? It optimizes the real objective and needs no surrogate. Its cost, however, grows with the number of greedy steps times the evaluation cost, and it yields no understanding of the coupling. We compare at matched and at unlimited evaluation budgets. If it wins at matched budgets, our method has no practical case beyond diagnosis.

Why not just use data-free structural scores like NSDS? They are nearly free and robust to 2.6 bits. They score static weights and cannot see how one layer's error reshapes another's inputs. If NSDS matches us on Qwen2.5-7B and Llama-2-7B at 3 bits, then cross-layer interactions matter less for dense LLMs than the Kronecker-Fisher coupling results imply. That would be worth knowing.

Proposed Experiments and Decisive Checks

We use two regimes: ResNet-50 and MobileNetV2 on ImageNet, plus BERT on SQuAD, for direct comparison with the Augmented Hessian numbers; and Llama-2-7B and Qwen2.5-7B, scored by WikiText-2 and C4 perplexity and zero-shot ARC, PIQA, HellaSwag, WinoGrande and BoolQ, at average budgets of 2.5, 3 and 4 bits. We train nothing; all allocations use post-training quantization with the same quantizer and calibration data. Mixtral-8x7B, compared against Q-Strata's reported WikiText-2 perplexity of 12.14 at 1.75 bits, is a stretch goal only if the dense results hold.

The first decisive check is predictive. On held-out random configurations, we report the Spearman correlation and relative error between predicted and measured $D(b)$ for the Fisher-only, additive, local-window ($k$ = 1, 2, 4, with and without hubs), sparse-all-pairs, and tensor-train surrogates. The locality claim survives only if the windowed-plus-hub model comes within a small margin of the unrestricted tensor-train surrogate.

The second is diagnostic. We plot interaction mass against depth distance and order at 4→3 and 4→2 drops. The hypothesis predicts rapid decay outside hubs, and a growing share of residual interaction at 2 bits concentrated in Output and Value projections. A flat curve falsifies locality.

The third is end-to-end. We report accuracy or perplexity at matched average bits against all baselines above, with three calibration seeds, alongside total GPU-hours including measurement. This check determines whether the measured structure turns into better compressed models, or only into a better understanding of them.