Compressed language models are judged almost entirely by perplexity, a measure of how surprised the model is by held-out text, and by multiple-choice quiz benchmarks. Perplexity is an average over every token, and most tokens are easy words like "the" and "of". A model could therefore lose the ability to make a verb agree with a subject twelve words back and still post a near-lossless perplexity score. This proposal measures that blind spot directly. We isolate the small set of tokens in ordinary text whose correct form depends on a distant word, and we track how 2–4 bit quantization (storing each weight in a few bits instead of sixteen) and structured pruning affect predictions on exactly those tokens. We then reweight the compression objective toward them, so the model keeps its long-distance grammar at the same memory cost.
The bet has two parts. First, aggressive compression damages non-local syntax much more than aggregate perplexity or commonsense quizzes show, and a perplexity restricted to long-dependency tokens predicts that damage far better. Second, the leading loss-guided quantizer can be fixed by changing one thing, which tokens its loss counts. This costs no extra bits and no expensive search loop.
Core Claim
Three bodies of evidence meet here. The perplexity–syntax literature shows the two quantities often come apart. Hu et al. (2020) ran 20 model–data combinations over 34 psycholinguistic test suites and found that perplexity and syntactic generalization frequently dissociate. Pérez-Mayos et al. (2021) likewise found that lower perplexity does not imply better targeted syntax. Hill et al. (2016) explains the mechanism: perplexity is dominated by frequent words. These papers motivate the proposal. All of them, however, study models that were trained differently. None asks what happens when a single trained model is compressed.
The compression literature supplies the method, and it has exactly this blind spot. GuidedQuant chooses quantization parameters using gradients of the model's end loss. On Llama-2-7B at 2 bits it brings WikiText-2 perplexity from 39.58 down to 8.83, but it evaluates only perplexity and commonsense multiple choice. AdaPruner and TACQ are evaluated the same way, on perplexity, quizzes, math, and SQL. None of them reports minimal-pair or agreement results.
The zero-shot literature defines where the damage would show up in practice. Frozen models already struggle on outputs with rigid structure. Li et al. (2025) report 10–21% zero-shot F1 for GPT-4o and Qwen-72B on semantic role labeling, the task of marking who did what to whom in a sentence. Multiple-choice suites cannot detect this kind of failure.
Our claim is specific and falsifiable. At 2–3 bits, loss-guided quantizers recover aggregate perplexity while long-dependency perplexity and minimal-pair accuracy on non-local phenomena lag behind. Weighting the guidance loss toward long-dependency tokens closes most of that gap at fixed bit-width and fixed calibration cost.
Research Program
The measurement. We run a dependency parser over natural evaluation text: the WikiText-2 and PTB test sets plus a held-out C4 slice. We mark every token that is the dependent end of a grammatical relation spanning at least $k$ intervening words. These include verbs agreeing with a distant subject, reflexives bound by a distant antecedent, and negative-polarity items licensed by a distant negation. We then define long-dependency perplexity, LD-PPL, as perplexity computed only over these marked tokens.
As a control we build a frequency-matched local set. It holds the same relation types with $k \le 1$, sampled so that the word-frequency distribution matches the long set. The point of the control is to separate "distance" from "rare word." Because the measurement uses natural text, it also answers the standard objection that template minimal pairs are artificial. We still report BLiMP, the Marvin and Linzen suites, and the Hu et al. test suites alongside it.
The objective. GuidedQuant weights each layer's reconstruction error using gradients of the end loss. We change only that loss to
$$\mathcal{L}_w = \frac{1}{\sum_t w_t}\sum_t w_t\,\ell_t, \qquad w_t = 1 + \lambda\,\mathbb{1}[t \in D_k],$$
where $\ell_t$ is the next-token loss at position $t$, $D_k$ is the set of long-dependency tokens in the calibration text, and $\lambda \ge 0$ controls how strongly they are emphasized. Setting $\lambda = 0$ recovers GuidedQuant exactly. Parsing the calibration set is a one-time CPU cost that is negligible next to collecting the gradients. Bit-width, group size, and inference kernels stay unchanged, so the deployed memory and latency are identical to those of the baseline.
The analysis. Across a grid of compressed variants, we test which proxy best predicts degradation in syntax and structured outputs: aggregate perplexity, commonsense accuracy, or LD-PPL. The grid covers GPTQ, GuidedQuant, TACQ, AQLM, ASVD, and AdaPruner at several bit-widths or pruning ratios. If LD-PPL is the strongest predictor, compression papers gain a cheap check that requires no hand-built test suites.
Baselines
The naive reference point is the uncompressed FP16 model. It sets the ceiling for every syntax and structure metric. Alongside it we run plain GPTQ with layer-wise squared-error reconstruction, which gives the "do nothing special" compressed point.
The strongest existing method is GuidedQuant at its own settings. It reports WikiText-2 perplexity 8.83 for Llama-2-7B at 2 bits, against 39.58 for the baseline it compares with. We rerun it with the public settings on the same calibration text, the same bit-widths, and the same gradient budget as our weighted variant, so the only difference is $\lambda$.
For circuit-preserving quantization we use TACQ. It keeps a small fraction of task-critical weights in 16 bits and reports recovering 96% of unquantized MMLU at 3.1 bits on Llama-3-8B-Instruct. It is the strongest competitor that already tries to protect specific abilities, so we run it at matched average bits, including the 16-bit outliers. For extreme compression we include AQLM, whose repository reports WikiText-2 perplexity 7.85 for a roughly 1-bit Llama-2-7B checkpoint.
For pruning we use AdaPruner, which reports 61.44% average commonsense accuracy on LLaMA-7B at 20% pruning. For low-rank compression we use ASVD.
No reported result was found on Lacuna for BLiMP, Marvin–Linzen, or LD-PPL on any compressed model. Our FP16 and baseline reruns therefore establish those numbers. The only published syntactic anchor in the file is GPT-2 at 83.0% overall on BLiMP, against a human ceiling of 88.6%.
Three ablations isolate the key ingredient. The first applies the same up-weighting to the frequency-matched local set. The second applies it to a random token set of equal size. The third applies it to the long-dependency set with $\lambda$ swept. If the long-dependency weighting wins only because it up-weights rare words, the local-set ablation will match it.
Beating GuidedQuant has a concrete meaning. At equal bits, the weighted variant must close at least half of the GuidedQuant-to-FP16 gap on the non-local BLiMP categories (islands, long-distance agreement, NPI licensing) and on LD-PPL. It must also keep aggregate WikiText-2 perplexity within 2% of GuidedQuant's value, and it must not reduce the commonsense average.
Why not just…?
Why not just calibrate on syntax-rich text or on minimal pairs instead of reweighting tokens? This is the most serious threat, and we test it directly. We run GuidedQuant and TACQ with calibration sets drawn from parsed text that is dense in long dependencies, and separately with BLiMP-style sentence pairs at matched token count. Our reason to expect reweighting to do better is the averaging problem from Hill et al.: even syntax-dense text is mostly function words, so the gradient signal stays dominated by easy tokens. Calibrating on templates also risks fitting their surface patterns. The GED probing study found that probes trained on learner essays degraded on Wikipedia edits, which is this kind of failure. If syntax-dense calibration alone matches our method, we will report that and recommend it, because it is simpler.
Why not just give TACQ syntactic data so it finds and protects "grammar circuits"? Shared Circuits for Shared Grammar does show that agreement uses localized attention-head circuits, which makes this a natural extension. The cost is that it spends extra 16-bit weights, and it localizes from minimal-pair prompts that may not cover the variety of natural text. We include TACQ calibrated on minimal pairs as a baseline at matched average bits. If it wins on BLiMP but loses on natural-text LD-PPL, that would indicate it protects the templates rather than the underlying ability.
Why not just fix syntax afterwards with a LoRA adapter on the quantized model? Li et al. (2025) show that small adapters, together with retrieval and self-correction, can rescue structural tasks. Their pipeline, however, needs labeled task data, extra inference passes, and a separate adapter for each task. Our method changes only the one-time compression step. We run a matched-budget adapter recovery on the same calibration text as a comparison. We also apply the Li et al. adapter pipeline on top of each compressed base to test whether better-preserved syntax in the base produces better adapted SRL.
Why not just use a bigger model at fewer bits? Scale does not remove structural failures. Li et al. show 72B-class frozen models still score 10–21% zero-shot SRL F1, and Hu et al. found that inductive bias outweighed data scale. Even so, we include a size–bits comparison at matched memory on Qwen2.5 sizes to check this.
Proposed Experiments
We use Llama-2-7B, to match the GuidedQuant setting, and Llama-3-8B-Instruct, to match TACQ, with Qwen2.5-7B and Qwen2.5-32B-Instruct as a scale check. We compress at 4, 3, and 2 bits, and prune at 20% and 40%.
Each variant is evaluated on aggregate perplexity (WikiText-2, PTB, C4), LD-PPL with its local and random controls, BLiMP overall and by phenomenon, Marvin–Linzen agreement across relative clauses, the Hu et al. syntactic-generalization score on BLLIP-style suites, the commonsense suite, and SRL on CoNLL-2009 and CoNLL-2012. SRL is run with the Li et al. pipeline on each compressed base, with boundary-error rate reported separately from F1. Memory, tokens per second, and calibration GPU-hours are reported for every method.
Three outcomes decide the proposal. The first is whether the dissociation exists. Across baseline-compressed variants with aggregate perplexity within 5% of FP16, do the non-local BLiMP categories fall by several points while local agreement holds? A null result here would be informative in its own right: compression would then be safe for syntax, and the rest of the program would become a reporting standard rather than a fix.
The second is whether LD-PPL predicts structure better than aggregate perplexity. Across all compressed variants, LD-PPL's rank correlation with BLiMP-non-local and SRL degradation should clearly exceed that of aggregate perplexity and commonsense accuracy.
The third is whether the long-dependency weighting is the active ingredient. The weighted GuidedQuant must meet the beating criterion stated above, and the frequency-matched local and random weightings must not. A further result would strengthen the claim: the per-layer quantization error saved by our weighting should concentrate in the middle-to-late layers where the GED probing study located agreement information. This would tie the gain to a specific mechanism and rule out a generic regularization effect.
References
- A Systematic Assessment of Syntactic Generalization in Neural Language Models
- How much pretraining data do language models need to learn syntax?
- The Goldilocks Principle: Reading Children's Books with Explicit Memory Representations Felix Hill
- GuidedQuant: Large Language Model Quantization via Exploiting End Loss Guidance
- Sample-aware Adaptive Structured Pruning for Large Language Models
- Task-Circuit Quantization: Leveraging Knowledge Localization and Interpretability for Compression
- LLMs Can Also Do Well! Breaking Barriers in Semantic Role Labeling via Large Language Models
- BLiMP: The Benchmark of Linguistic Minimal Pairs for English
- Targeted Syntactic Evaluation of Language Models
- ASVD: Activation-aware Singular Value Decomposition for Compressing Large Language Models
- Probing for targeted syntactic knowledge through grammatical error detection
- Shared Circuits for Shared Grammar: Tracing Subject-Verb Agreement Across Languages