Language models are usually described as learning grammar early and cheaply, and struggling only with rare facts. This proposal tests whether that picture hides a third case. Some syntactic patterns are rare too: center-embedded relative clauses, agreement across an intervening clause, and certain extraction structures. If these are learned the way rare facts are, then the right way to measure a model's training data for them is the number of times each construction occurs, not the number of tokens. If that count has to pass a threshold before the model gets the construction right, adding more raw text is a wasteful way to fix the problem.
The tool that makes this testable comes from an older line of work. Treebank search engines index the tree inside each parsed sentence so that structural patterns can be found in seconds across tens of millions of sentences. Here such an index is used for a different purpose. It counts every occurrence of a construction in the pretraining corpus, deletes or adds occurrences on demand, and feeds the training data loader. In two sentences: we use a structural search index over a parsed pretraining corpus to measure and manipulate how often each syntactic construction is seen, and test whether per-construction accuracy follows a frequency threshold that shifts with model size. If it does, the same index can reschedule training data so rare constructions cross that threshold at a fixed token budget.
Core Claim
Understanding LLM Behaviors via Compression splits language into a parametric "syntax" part, which has finite complexity and is learned first, and a non-parametric "knowledge" part, which follows Zipf's law and is learned in order of frequency. For a 7.2M-parameter model it finds a sharp cutoff: facts seen fewer than 508 times were consistently hallucinated. That work treats syntax as one abstract block and never looks at explicit trees. Our claim is that syntax is not one block. The head of the construction distribution behaves like their syntax component, but the long tail behaves like their knowledge component and has its own occurrence threshold. Concretely, let $n_c$ be the number of training occurrences of construction $c$, let $M$ be the model's parameter count, and let $\mathrm{Acc}_c$ be the model's accuracy on held-out minimal pairs for $c$. We predict
$$\mathrm{Acc}_c \approx 0.5 + 0.5\,\sigma\big(a\,(\log n_c - \log \tau(M))\big),$$
where $\sigma$ is the logistic function, $a$ controls how sharp the transition is, and $\tau(M)$ is a size-dependent threshold that falls as $M$ grows. The 0.5 floor is chance on a two-way minimal pair. The logistic-in-log-count form is the shape the compression paper's threshold finding implies.
The claim fails in one of two ways. Accuracy could depend on total tokens and not on $n_c$, so constructions are free riders on general syntax. Or $\log n_c$ could predict accuracy no better than surface frequency of the construction's words. This is the same move that Towards Neural Scaling Laws on Graphs made when it showed graph scaling is fit better by edge count ($R^2 = 0.94$) than by graph count ($R^2 = 0.84$). We propose that the right unit of data for syntax is the occurrence of a structural relation, not the token or the sentence.
Research Program
The external-structure side reuses existing systems. Fast Query for Large Treebanks and Fangorn treat each parsed sentence as a separate document in a Lucene inverted index. Each tree node gets four integer coordinates (left, right, depth, parent), so ancestor and sibling tests become integer comparisons. A //NP query over 26.5M Wikipedia sentences took 15.53 s, against 89.35 s for TGrep2, a standard tree-search tool. These systems were built only for human linguists running queries; nobody has connected them to training. We take the method unchanged and add two things.
The first is regular-expression matching over node labels. Fangorn treats labels such as NP-SBJ-1 as atomic strings, which is a limitation the authors note. The second is a data-loader interface. It returns posting lists of sentence IDs per construction and supports three operations under a fixed token budget: delete (remove all but $k$ occurrences of $c$), inject (repeat or draw additional matching sentences), and reweight. We stay with constituency trees because Fangorn's span-containment scheme cannot represent dependency graphs. A dependency extension is out of scope.
The internal-structure side is an inventory of roughly 40 construction queries in this extended query language. They span frequencies from very common to a few hundred occurrences: subject–verb agreement with and without an intervening prepositional phrase or relative clause, single and double center embedding, object versus subject relatives, wh-extraction from embedded clauses, and similar patterns. For each construction we build held-out minimal pairs automatically from trees outside the training split. The grammatical sentence is paired with a version perturbed along the construction's critical relation, for example agreement flipped on the verb whose subject sits across the relative clause. Pairs are filtered so that the two sentences have matched word-level frequency.
The analysis has three stages. Stage one is observational: train models of several sizes on the natural corpus and fit the equation above across constructions. Stage two is causal and is what separates this from a correlation study. For a randomly chosen half of the constructions, the index removes occurrences down to a set dose $n_c \in \{0, 30, 100, 300, 1000, 3000\}$. The removed tokens are replaced by token-matched random sentences, so the total budget is fixed. Stage three is the application: index-driven rebalancing that raises every tail construction above the $\tau(M)$ estimated in stage two, again at a fixed token budget.
Baselines
The naive reference point is training on the natural corpus in its natural order with no intervention, at matched tokens, parameters, and optimizer settings. Every arm is reported against this number on per-construction minimal-pair accuracy, macro-averaged over tail constructions. Overall perplexity on a held-out split checks that nothing else broke. We found no reported result on Lacuna for construction-level minimal-pair accuracy as a function of construction count, so the reference numbers come from our own runs. The closest reported figure is the 508-occurrence fact threshold at 7.2M parameters from Understanding LLM Behaviors via Compression. We treat it as a scale anchor for where $\tau(M)$ might sit at small $M$, not as a target.
The strongest competitor is simply training on more raw tokens, the lesson of Scaling to Very Very Large Corpora. Banko and Brill found log-linear gains to a billion words and found that data volume swamps algorithm choice. We run it as a model of the same size trained on 2× and 4× more natural tokens. The rebalanced 1× model beats it only if its tail-construction accuracy matches or exceeds the 2× model's while head-construction accuracy and perplexity stay within seed noise.
The ablation that isolates the key ingredient, structural rather than surface access to the corpus, is surface-pattern rebalancing. Here we upsample sentences that match word-level or part-of-speech n-gram proxies for each construction (for instance "who/that … verb" for relative clauses), using a plain word index with the same token budget and the same number of selected sentences. Random upsampling of an equal number of sentences controls for repetition alone. All arms share seeds (three per cell), the learning-rate schedule, and a tuning budget fixed on the natural-distribution run and not re-tuned per arm.
Why not just…?
Why not just train on more data? That is what Banko and Brill would predict works, and it may work for the head of the distribution. Under Zipf-like construction frequencies, though, doubling tokens doubles $n_c$ for every construction. If $\tau(M)$ is real, lifting a construction seen 50 times past a threshold of several hundred takes roughly an order of magnitude more data, while targeted injection costs a fraction of a percent of the budget. The 2×/4× arm measures this directly. If tail accuracy rises with total tokens faster than $n_c$ alone predicts, our claim is wrong and that result will be reported as such.
Why not just match surface strings instead of trees? This is the real threat. For constructions with reliable lexical markers, an n-gram proxy may find most occurrences, and then the treebank index adds cost without benefit. The surface-pattern ablation settles it. We also report, per construction, the precision and recall of the surface proxy against the tree query. The prediction is that the structural index helps exactly where proxy precision is low, as with agreement across clauses or double center embedding, and does nothing where the proxy is accurate. If the advantage does not track proxy precision, the structural index is unnecessary and we will say so.
Why not just pretrain on synthetic grammar, as SIP does? SIP instills a structural bias by pretraining on 40,000 synthetic finite-state transducers. It raised length generalization from 37.8% to 94.8%. Finite-state machines cannot express nesting, however, and the authors list pushdown transducers as future work. A synthetic context-free pretraining arm is a sensible future comparison, but it tests whether a generic bias toward nested structure helps. It does not test whether specific natural constructions need a specific number of occurrences, which is the question here. The two are complementary, and we do not claim synthetic data would fail.
Why not just tune domain mixing weights? Domain weights act on whole documents. Their effect on a particular construction is indirect and confounded with topic. Sentence-level selection by structure is a strictly finer-grained version of the same operation. The random-upsampling and surface arms bound how much of any gain is just the effect of changing the mixture.
Proposed Experiments
The corpus is the parsed English Wikipedia collection of about 26.5M Penn-Treebank-format sentences used by Fast Query for Large Treebanks. To test robustness to domain, we add a parsed sample of FineWeb. Decoder-only models are trained at four sizes from roughly 10M to 300M parameters. The smallest overlaps the compression paper's 7.2M setting, and the largest matches SIP's scale. Each construction's held-out minimal pairs (at least 500 per construction) come from a disjoint 5% of trees, and all occurrence counts are computed only on the training split.
The first decisive check is the dose–response experiment. If accuracy on manipulated constructions follows a logistic curve in $\log n_c$, with a fitted $\tau(M)$ that falls as $M$ grows, while unmanipulated constructions in the same models stay unchanged, frequency-thresholded syntax is established causally. If the curves are flat, or track total tokens, the claim is false.
The second check is the observational fit. Construction count should explain per-construction accuracy better than the summed word frequency of the construction's tokens. This is our analogue of the edge-versus-graph-count comparison.
The third is the application at a fixed token budget. Index-driven rebalancing should beat the natural distribution and the surface and random arms on tail accuracy. It should approach or exceed the 2× natural-data model, and leave perplexity and head-construction accuracy unchanged.
Finally, we record the index's cost: build time, query latency per construction, and loader overhead as a fraction of training time. Then the result can be stated honestly as accuracy gained per unit of wall-clock time, not accuracy alone.
References
- Understanding LLM Behaviors via Compression: Data Generation, Knowledge Acquisition and Scaling Laws
- Towards Neural Scaling Laws on Graphs
- Fast Query for Large Treebanks
- Fangorn: A System for Querying very large Treebanks
- Scaling to Very Very Large Corpora for Natural Language Disambiguation
- SIP: Injecting a Structural Inductive Bias into a Seq2Seq Model by Simulation