Latency Backdoors in Non-Uniform Quantized Inference

To serve large models on cheap hardware, practitioners compress weights from 16-bit floats down to 2–4 bits. Because weight values cluster in a bell shape, an evenly spaced (uniform) grid wastes most of its levels on the sparse tails, so advanced methods use non-uniform quantization: the levels are placed to match the actual distribution, via learned grids, power-of-two sums, or vector codebooks. The price is a "hardware tax." Standard chips multiply integers on a linear grid, so non-uniform values must be recovered through lookup tables, sub-byte unpacking (bit-shifting and masking odd widths into byte-aligned registers), and high-precision side paths for outlier values — all of which cost latency and energy (FLUTE; UNIQ; OliVe). Separately, a line of work on backdoors — models that behave normally until an attacker-chosen trigger input appears — has shown that quantization itself can be the trigger: models can be built to be benign in full precision and malicious once compressed. But every existing quantization-activated attack (ACL; Widening the Gap; Qu-ANTI-zation; Quantization Backdoors) targets uniform integer rounding and attacks predictions.

This proposal moves the quantization-activated backdoor to the latency axis and aims it squarely at the non-uniform hardware tax. The claim is that the extra execution machinery non-uniform quantization requires is input-dependent, and that dependence is an attack surface no one has examined. A backdoor can be trained so the model is behaviorally and latency-normal in full precision and on clean inputs, but trigger inputs route computation through the slowest parts of the compressed execution path — the outlier side channels, the table lookups, the unpacking logic — producing latency and energy spikes only on the quantized deployment. If this works, the hardware tax is not merely a performance cost but a security liability that full-precision auditing is structurally blind to.

Core Claim

Existing "sponge" attacks already poison weights to inflate inference latency (Energy-latency attacks via sponge poisoning; SkipSponge), but they slow the model down everywhere, on every input and at every precision, so a latency profile of the full-precision checkpoint exposes them. Existing quantization-activated attacks are precision-gated but change outputs, not speed, and are confined to uniform grids. The proposal sits in the empty quadrant: precision-gated and latency-targeted. Formally, the attack optimizes a poison that maximizes the amplification

$$\Lambda = \frac{T_{\text{quant}}(\text{trigger input})}{T_{\text{quant}}(\text{clean input})}$$

subject to $T_{\text{fp16}}(\text{trigger}) \approx T_{\text{fp16}}(\text{clean})$ (stealth at full precision) and unchanged clean accuracy after quantization. The mechanism is routing, not sabotage: mixed-precision schemes that isolate outliers in FP16/INT8 (Value-aware Quantization; OliVe) create a two-speed datapath with dynamic branching and memory divergence — the trigger is trained to maximally engage the slow lane. In lookup-table kernels (FLUTE; QuIP$\#$; DeepGEMM), the trigger steers activations toward code patterns whose dequantization is most expensive.

Research Program

The program has three parts, ordered so the first is a go/no-go gate. First, a latency-surface audit: measure, per input, the inference latency of state-of-the-art non-uniform pipelines — FLUTE on GPU serving, QuIP$\#$'s E8-lattice path, the AQLM codebase (Vahe1994/AQLM) with arbitrary 8-dimensional codebooks, and DeepGEMM's SIMD tables on CPU — and compare the input-to-input latency spread against FP16 matrix multiply and uniform INT8. This must be done honestly: FLUTE's authors duplicated their tables in shared memory precisely to kill bank conflicts, so the residual input-dependence may be small. If non-uniform kernels show no more input-conditional latency variance than uniform INT8, the attack idea fails and we report that as a negative result — itself useful, as it would certify these kernels against this class.

Second, attack construction on the most promising surface the audit identifies. The primary candidate is the outlier side path: following the two-stage recipe of Quantization Backdoors — embed the circuit at full precision, then fine-tune to remove full-precision sensitivity — we poison so that trigger inputs (e.g., a rare Unicode homograph token, as in Hidden Backdoors) drive outlier channels the quantizer kept in high precision, while clean inputs route through the fast path. The training objective is the amplification $\Lambda$ above, estimated with a differentiable latency surrogate (counts of slow-path engagements) plus a penalty on FP16 latency change and clean loss.

Third, defense implications, including one side experiment that resolves an open contradiction in the literature: Widening the Gap breaks group-wise uniform quantization by injecting giant outliers, while QuIP$\#$'s Randomized Hadamard Transform exists precisely to scatter outliers across coordinates. Whether incoherence processing is an intrinsic defense against outlier injection is unstudied; we rerun their attack against QuIP$\#$ and report the answer either way. We also test latency-level defenses (profiling with random inputs; coverage-guided latency fuzzing) and note that standard backdoor scans are already known to collapse under quantization — Quantization Blindspots found the aggregate detection rate of five defenses fell to 0% at INT8 — so output-based auditing cannot be relied on.

Baselines

The "do nothing" reference is the clean compressed model: QuIP$\#$ serves a 2-bit Llama-2 7B at 106.3 tokens/sec on an RTX 4090 versus 33.1 tokens/sec for the FP16 baseline (QuIP$\#$), and FLUTE's kernels run 2–4x faster than standard matrix-multiply kernels at batch sizes below 32 (FLUTE). We will first measure the natural input-to-input latency spread of these clean systems; the attack counts as real only if trigger latency exceeds that natural spread by a wide margin and, ideally, drags trigger-input throughput below the FP16 floor of 33.1 tokens/sec — erasing the entire point of compressing. The strongest existing competitor is sponge poisoning (Energy-latency attacks via sponge poisoning; SkipSponge), the established weight-poisoning latency attack; no reported result found on Lacuna for its latency amplification on transformer serving, so we will run it ourselves under the same models, data, and tuning budget, deployed both at FP16 and quantized. Beating it means achieving comparable or larger trigger latency on the quantized deployment while remaining within noise on the FP16 checkpoint, where sponge poisoning's slowdown should still be visible. The key ablation deploys the same poisoned model through uniform INT8/INT4 (GPTQ-style) and through the non-uniform pipeline: if latency spikes appear equally under uniform quantization, the "non-uniform overhead" ingredient is doing no work and the claim is falsified. For the side experiment, the baseline is Widening the Gap, which reports post-quantization attack success rate consistently above 95% (with full-precision ASR below 5%) on Llama 3.1-8B, Qwen 2.5-7B, and Mistral-7B against GPTQ, AWQ, and GGUF; we rerun it unchanged against QuIP$\#$ with the same tuning budget, and Hadamard incoherence counts as a defense only if that number drops sharply.

Why not just…?

Why not just run existing sponge poisoning on the quantized model? Because sponge poisoning is not precision-gated: it inflates latency by saturating computation with extra work, which shows up at FP16 too, so a pre-deployment latency profile catches it. Our attack must be invisible at FP16 by construction. The honest version of this objection is a constrained sponge that also penalizes FP16 latency — that is a real threat, and the ablation above settles it: if the constrained sponge achieves the same amplification on uniform INT8 as on non-uniform kernels, then nothing is specific to non-uniform overhead and we say so plainly.

Why not just audit latency before deployment, or fuzz the deployed model? Random-input profiling will not find a trigger chosen by the attacker to be rare and specific — the same reason Quantization Backdoors evaded Neural Cleanse, STRIP, and ABS at full precision. Coverage-guided latency fuzzing is the serious version, and we will build it as the strongest defense candidate rather than argue around it; if it reliably finds trigger inputs from latency anomalies alone, that bounds the attack's practical impact and is a finding worth publishing.

Why not just stick to prediction-axis attacks and port them to non-uniform quantization? That is the obvious intersection study, and it is genuinely undone — every attack in the literature is uniform-only — but it is an incremental port. We include its most interesting instance (Outlier Injection versus Hadamard incoherence) as a decisive side experiment, because its outcome determines whether attackers even need new mechanisms on the output axis. The latency axis is where this proposal earns its keep: no cited work, on either side of the intersection, touches input-conditional execution cost in non-uniform kernels.

Why not just avoid non-uniform quantization and deploy uniform INT8? Deployers accept the hardware tax because it buys accuracy at extreme compression — QuIP$\#$ reaches WikiText-2 perplexity 4.16 at 2 bits on Llama-2 70B versus 8.17 for its predecessor (QuIP$\#$), and AQLM's 1-bit Llama-2 7B reaches 7.85 (Vahe1994/AQLM) — and because uniform quantization already carries its own activated backdoors (Qu-ANTI-zation). Retreating to INT8 trades one attack surface for another.

Proposed Experiments and Decisive Checks

Models and systems: Llama-2 7B and 70B through the QuIP$\#$ pipeline, Llama-3.1 8B through FLUTE integrated into vLLM, and AQLM's codebase for arbitrary-codebook ablations, on an RTX 4090-class GPU to match the reported 106.3/33.1 tokens/sec figures; DeepGEMM extends the study to CPU. Utility is verified with the protocol of Widening the Gap — retain over 90% of original performance on MMLU, ARC-Challenge, HellaSwag, HumanEval, and GSM8K — plus WikiText-2 perplexity. The primary metric is $\Lambda$ with its FP16-stealth constraint; secondary metrics are energy per trigger token and the fraction of trigger inputs whose throughput falls below the FP16 floor. Decisive checks, in order: the audit must show non-uniform kernels with materially larger input-conditional latency variance than uniform INT8 (else stop); the attack must beat sponge poisoning under matched budgets while staying FP16-latency-clean; the uniform-deployment ablation must show the amplification collapse; and latency fuzzing must be given a fair chance to break stealth. The side experiment on Hadamard incoherence versus outlier injection reports whichever outcome holds. Together these either establish the hardware tax as a new backdoor surface or certify the current kernel generation against it.