Large language models can pick up a new pattern from a handful of examples placed in the prompt, a skill called in-context learning. Nobody knows whether this skill extends to grammar. The question is whether a frozen model, shown a few dozen unlabeled sentences of an unfamiliar language, builds an internal picture of how words in that language group into phrases. If it does, a second question follows: does that picture survive when the tokenizer chops the language's words into many arbitrary pieces? The tokenizer is the front-end that splits text into the model's units, usually frequency-based fragments called subwords. For low-resource and morphologically rich languages, those fragments rarely line up with real word parts.
We propose measuring in-context grammar learning directly, inside the model, on artificial languages whose grammar we control. We then vary how the tokenizer fragments their words while keeping the grammar identical. In two sentences, the idea is this. We read out the tree structure a frozen model assigns to a sentence after it has seen unlabeled examples of a made-up language, and check whether that structure matches the true grammar. We then ask whether the well-documented accuracy penalty on heavily fragmented languages comes from long sequences, from inconsistent splitting of the same word, or from tokens that straddle word boundaries.
Core Claim
Two existing findings sit uneasily together. The Token Tax shows that token fertility, meaning tokens per word, predicts up to 50% of the accuracy variance of ten models on AfriMMLU across 16 African languages. That study only uses multiple-choice question answering and offers no mechanism. Pan et al. separate two things a prompt can do. It can trigger task recognition, which retrieves a skill learned in pretraining. It can also drive task learning, which acquires a new input-output mapping from the examples. Their key finding is that task learning is fragile on structurally complex tasks. Meanwhile Truong et al. show that models handle affixal negation in familiar words but fail on nonce words, and recover when hyphens force clean splits.
Our hypothesis joins these findings. Learning a grammar in context depends on the model tracking which units co-occur, the kind of local statistics that Deep networks learn to parse uniform-depth context-free languages from local statistics shows suffice for learning grammar in the weights. That tracking is damaged most by segmentation inconsistency: the same word surfacing as different token sequences in different contexts. Raw sequence length and boundary-straddling tokens matter less. A correct hypothesis predicts a specific pattern. Holding total prompt length fixed, inconsistent segmentation should depress recovered tree accuracy far more than consistent multi-token segmentation of the same length.
The confound is the reason this question has stayed open. Natural languages offer no way to separate fertility from consistency or from pretraining priors. Artificial languages with nonce vocabularies remove the priors, which forces the model into task learning rather than recognition. Choosing the nonce strings lets us set each tokenization property independently.
Research Program
We first build a family of probabilistic context-free languages. Each has around 10 to 30 word categories, recursive phrase rules, and constructions where a hierarchical rule and a linear "count the words" rule make different predictions, such as agreement across an embedded clause. Words are nonce strings. Using the actual tokenizers of Llama-3-8B, Qwen-2.5, and the Pythia suite, we select strings so that each language variant lands in one cell of a factorial design. The first factor is mean fertility: one, two, or four tokens per word. The second is consistency: each word always splits the same way, or it splits differently depending on its neighbors, which we produce by exploiting whitespace and punctuation merges. The third is whether any token crosses a word boundary. The grammar and the sampled sentences are identical across cells, so only the token stream changes.
Next we extract the model's implicit structure without training any probe. We adapt Perturbed Masking, which builds an "impact matrix" for BERT by measuring how masking one word changes the model's representation of another, and then decodes a tree from it. Our version targets causal models. We replace word $i$ in a query sentence with another word of the same category and record the change in the model's hidden state at word $j>i$. Each word is pooled over all of its tokens, so every tokenization cell is scored on the same word-level matrix. We decode undirected dependency trees with a maximum spanning tree and constituency brackets with top-down splitting, sweeping layers around the intermediate "sweet spot" that In-Context Learning Creates Task Vectors and One Task Vector is not Enough both place near layer 15 in 8B-scale Llama models.
The in-context learning signal is the gain in tree accuracy as the number of unlabeled prompt sentences grows from 0 to 64. This gain is reported alongside three behavioral measures:
- minimal-pair preference between grammatical and ungrammatical continuations;
- hierarchical-versus-linear generalization on held-out constructions;
- the grammaticality of sampled sentences under the true grammar.
A secondary, mechanistic arm asks where the learned structure lives. We patch activations from a grammar-bearing prompt into a prompt-free run, either as one vector or as per-position states. One Task Vector is not Enough found that structured outputs such as JSON need distributed sub-task vectors. We test whether grammar does too, and whether fragmentation spreads it further.
Finally we test a fix that requires no oracle. Truong et al. relied on hyphens placed with knowledge of the true morphology. We instead re-segment the prompt so that each recurring surface form is always tokenized identically, by forcing a canonical split for any string seen at least twice in the prompt. This addresses inconsistency without knowing where the morphemes are. If consistency is the culprit, this cheap step should recover most of the loss.
Baselines
The naive reference point is the same frozen model with zero prompt sentences. Its impact-matrix trees over nonce words should reflect only position and token-shape cues, so this run tells us how much structure is extracted with no learning at all. We pair it with right-branching and random-tree baselines computed on the same query sentences.
For trees read out of a frozen pretrained model, the strongest existing method is Perturbed Masking on BERT, which reports 58.6 UAS on WSJ10 for unsupervised dependency parsing. We run our causal adaptation on the same WSJ10 sentences, scored against the same word-level conventions, with and without other WSJ sentences in the prompt. Matching or exceeding 58.6 shows that the extraction method is sound. Gains beyond it from added context show in-context benefit on natural English.
For trees from models trained specifically to induce structure, the strongest numbers are StructFormer at 54.0 UF1 and 46.2 UAS on WSJ test, and Tree Transformer at 52.0 F1. We cite these only to locate English performance, since those models train in the weights rather than learning in context.
On the artificial languages, the real competitor is learning in the weights. We train a small transformer from scratch on exactly the prompt sentences, following the setup of Deep networks learn to parse uniform-depth context-free languages from local statistics and its code jackparley/learn_to_parse. It gets the same tokenization cells and a matched hyperparameter sweep. We extract its trees with the same impact matrix. No reported result found on Lacuna for this baseline on our grammars, so we produce the number ourselves.
The ablation that isolates our key ingredient is the length-matched consistency contrast. It compares consistent two-token words against inconsistent two-token words at identical total token counts. Our claim survives only if inconsistency costs substantially more tree accuracy than consistent fragmentation does, across at least two model families.
Why not just…?
Why not just ask the model to output bracketed parses? Generated brackets mix grammar knowledge with the model's ability to follow a bracket-output format. One Task Vector is not Enough shows that structured-output formats are precisely where in-context learning is weakest. LLM Dependency Parsing with In-Context Rules needed labeled trees and still trailed supervised parsers, with its gains vanishing by five shots. We will still run generated parsing, prompted with a definition of constituency but no labeled examples. If it tracks the impact-matrix results cell by cell, that is welcome convergent evidence. Where the two diverge, we report the internal measure as the cleaner one.
Why not just use minimal-pair surprisal, without extracting trees? This alternative is a real threat, since minimal pairs are cheap and well understood. Minimal pairs, however, cannot tell whether the model learned a hierarchical rule or a linear shortcut unless the pairs are built to pull the two apart. They also give no account of which attachments fragmentation breaks. We therefore include hierarchical-versus-linear minimal pairs as a co-primary measure. If they show the same consistency-versus-length pattern as the trees, the trees add diagnosis rather than discovery, and we will say so.
Why not just switch to byte-level models and avoid tokenization entirely? The Efficiency Gap in Byte Modeling documents a large compute penalty for byte-level modeling, and The Token Tax prices fertility in training cost. Bytes also do not test our hypothesis, since they are perfectly consistent but maximally long. That combination makes them a useful extreme cell, so we include a byte-level model where one is available.
Proposed Experiments
The primary experiment fills the full factorial on Llama-3-8B and Qwen-2.5 for 64 sampled grammars. It reports the gain in undirected attachment accuracy and in bracket F1 from in-context exposure, along with the three behavioral measures, all as a function of fertility, consistency, and boundary crossing. Scaling uses Pythia models from 160M to 12B parameters. Pan et al. found that task learning emerges only with scale, so we predict that the consistency penalty appears only once in-context grammar learning itself appears.
Two checks are decisive. The first is the length-matched consistency contrast described under Baselines. The second is the canonical re-segmentation fix. If it closes most of the inconsistency gap on artificial languages, we apply it to the low-resource Universal Dependencies languages used by LLM Dependency Parsing with In-Context Rules, such as Yoruba, Bambara, and Erzya. There we measure unlabeled attachment from impact matrices with and without re-segmentation; no reported result found on Lacuna for that paper's exact scores to compare against. The hypothesis is falsified if matched-length inconsistency costs no more than consistent fragmentation. It is also falsified if in-context exposure yields no tree gain over the zero-sentence reference at any scale. In that case, the token tax on structure would be a matter of sequence length rather than broken learning.
References
- The Token Tax: Systematic Bias in Multilingual Tokenization
- What In-Context Learning "Learns" In-Context: Disentangling Task Recognition and Task Learning
- Revisiting subword tokenization: A case study on affixal negation in large language models
- Deep networks learn to parse uniform-depth context-free languages from local statistics
- Perturbed Masking: Parameter-free Probing for Analyzing and Interpreting BERT
- In-Context Learning Creates Task Vectors
- One Task Vector is not Enough: A Large-Scale Study for In-Context Learning
- StructFormer: Joint Unsupervised Induction of Dependency and Constituency Structure from Masked Language Modeling
- Tree Transformer: Integrating Tree Structures into Self-Attention
- LLM Dependency Parsing with In-Context Rules
- The Efficiency Gap in Byte Modeling