Language models copy constantly. They repeat a name that appeared earlier, finish a phrase they have already seen, or pull a rare term out of a document. The standard account of how Transformers learn this behavior is the "induction head": a two-layer circuit in which the first layer tags each token with the identity of the token before it, and the second layer finds the earlier occurrence of the current token and reads out what followed it. Theory says one ordinary attention layer cannot do this. Experiments show that two layers learn it only after a long, flat training plateau. This proposal asks a narrower question about what the second layer and the long plateau are actually paying for. We build a one-layer model in which the "previous token" tag is hardwired into the attention keys. We then compare two ways of producing the answer. In the first, the copied token's identity is pushed through the usual value projection and output vocabulary layer. In the second, a pointer copies the token directly: probability mass lands on the vocabulary entry sitting at the attended position, in the manner of the pointer-generator decoders used in summarization.
The hypothesis is that the bottleneck in shallow copying is mostly in decoding the answer, not in finding it. Locating the right earlier position only requires the model to tell apart the roughly $N$ tokens in the context, which should need width on the order of $\log N$. Writing the answer's identity through a $d$-dimensional vector and back out to a vocabulary of size $V$ is a different matter: the known gradient-descent analysis requires $d$ to grow like $\sqrt{V}$. If the pointer route removes the decoding cost, then a single layer of very small width plus a scalar gate (about $d+1$ extra parameters) should match two-layer models on associative recall and train without a plateau. Scaling width against $V$ and $N$ separately settles this cleanly.
Core Claim
Consider a single causal attention layer over tokens $w_1,\dots,w_t$. The query comes from the current token $w_t$, and the key at position $i$ is computed from the preceding token $w_{i-1}$, so the "shift" that an induction head's first layer would have to learn is fixed by construction. The output is a pointer-generator mixture
$$P(w) = p_{gen}\,P_{vocab}(w) + (1-p_{gen})\sum_{i\,:\,w_i = w} a_i,$$
where $a_i$ is the attention weight on position $i$, $P_{vocab}$ is the ordinary softmax over the vocabulary computed from the layer's output, and $p_{gen}\in[0,1]$ is a gate computed by a single linear map plus sigmoid from the layer output. The claim has three parts. First, for in-context associative recall, this model succeeds at width that grows with $\log N$ and is roughly flat in $V$. A matched one-layer model that uses the same shifted keys but must emit the answer through $P_{vocab}$ needs width that grows with $V$. Second, the pointer model trains without the plateau-then-jump dynamics seen in two-layer models, including in the regime where two-layer models get stuck. Third, when copy and non-copy targets are mixed, the gate is the component that breaks. An easy-to-hard schedule over the fraction of copyable targets is the cheapest way to keep it calibrated.
The first part is where most of the risk sits, and it is stated carefully. Even with random codes, decoding $V$ identities from $d$ dimensions can in principle be done with $d$ on the order of $\log V$. The $\sqrt{V}$ figure is a learnability bound for gradient training from Learning to Recall with Transformers Beyond Orthogonal Embeddings, which analyzes early training dynamics of a one-layer model whose vocabulary is far larger than its width. So the separation we predict concerns models trained by gradient descent, not what such models can represent, and it is an empirical claim this project tests rather than assumes.
Where This Sits
Two papers motivate the work by establishing the depth barrier and its training cost. How Transformers Implement Induction Heads proves that two layers are the minimal standard construction. It traces the training plateau to a separation of speeds: positional parameters grow quickly, while dot-product parameters grow slowly. The mechanistic basis of data dependence and abrupt learning shows that a two-layer attention model reaches induction only through an intrinsic stepping stone, a phase of guessing any label seen in context. When the number of labels equals the context length, removing that stepping stone, training gets trapped. Our design hardwires the fast positional half. If the plateau persists anyway, the slow half, matching plus decoding, is the culprit, and the pointer ablation tells us which of the two.
A second group constrains the design. Separations in the Representational Capabilities of Transformers and Recurrent Architectures shows that one layer of width $O(\log N)$ performs exact index lookup, which is the kind of position-finding our pointer relies on. The same paper shows that one layer needs linear width for string equality and nearest-neighbor tasks. Representational Strengths and Limitations of Transformers proves a single head handles pairwise matching but not triple matching. We therefore restrict the claim to single-hop recall and do not suggest the pointer addresses multi-hop relations; the open-problems section of the file notes that pointers cannot dereference indices in one step.
A third group supplies the mechanism and shows it is cheap. A Copy-Augmented Generative Model for Open-Domain Question Answering reuses a 12-layer T5 decoder's cross-attention as the copy distribution, adding 1,537 parameters. LargePiG extracts a pointer from frozen deep language models, setting its gate from the disagreement between intermediate and final layers. That depends on depth and cannot exist in one layer. Both treat the pointer as an add-on to deep networks. Neither asks whether pointing changes how much depth or width copying requires, which is our question. Simple and Effective Curriculum Pointer-Generator Networks trained a pointer decoder with an "answerability" curriculum, moving from examples where the answer is in the text to examples where it is not. Its ablation found that ordering mattered more than a chunk-length curriculum. Our gate curriculum is the synthetic, controllable analogue of that answerability schedule. Pointer-Generator Networks for Low-Resource Machine Translation: Don't Copy That! is the relevant warning: an uncontrolled gate copies when it should not. This is why we measure gate calibration rather than only accuracy. Finally, Induction Heads Interpolate N-Grams interprets soft attention weights and the beginning-of-sequence sink as n-gram smoothing. We use this to predict how our pointer should behave when the exact match is absent: it should back off toward the sink, and the gate should shift toward generation.
Research Program
The core object is the one-layer shifted-key pointer model described above. We derive a simple width bound for its matching step, using the random-embedding concentration argument behind the index-lookup result, and test that bound empirically. Around it we place four comparison models with matched training. The first is the same shifted-key layer emitting only through $P_{vocab}$, which isolates the output route. The second is a one-layer pointer without the shift, a sanity check that should fail at induction because it can only copy the current token. The third is a standard two-layer attention-only Transformer. The fourth is that two-layer model with the same pointer head attached to its second layer.
Width is swept separately against vocabulary size $V$ and context length $N$. We then fit the smallest width that reaches 99% recall accuracy as a function of each, which gives the separation in the core claim a slope we can measure. Training dynamics are tracked by the length of the loss plateau and by attention-to-correct-position over time. The gate is studied on mixed tasks where some targets are copyable and others follow a fixed bigram rule. We compare uniform mixing against a schedule that starts fully copyable and anneals the non-copyable fraction up. We also track forgetting of the copy mode, because the curriculum literature in the file flags this as the standard failure when moving from easy to hard data.
Baselines
The do-nothing reference is a one-layer attention model with ordinary keys and a vocabulary softmax. Theory says it cannot perform induction, and we expect chance-level recall on first-time associations. It is run with the same data and step budget as everything else. It exists to confirm that the task truly requires the circuit.
The strongest existing method is the standard two-layer attention-only Transformer of How Transformers Implement Induction Heads, with relative positional encoding, and also run on the label-classification task of The mechanistic basis of data dependence and abrupt learning. It is the strongest competitor because it is the accepted minimal architecture that learns copying from scratch. No reported result found on Lacuna for its minimal width or accuracy on synthetic associative recall. The file reports qualitative plateau-then-jump behavior on Wikitext-2, and for Tiny Shakespeare it reports that absolute-position models fail to attend to previous tokens beyond 100–128 tokens while relative-position models stay stable, per Rethinking Associative Memory Mechanism in Induction Head. For fairness, every model gets the same tokens, the same optimizer, a learning-rate sweep of equal size, and five seeds. Width is the swept variable, and both total parameter count and compute are reported. Beating this baseline means one of two things. Either the pointer model reaches 99% recall at a width the two-layer model cannot reach at any tested $V$ above a few thousand, or it reaches the same accuracy in a training run with no measurable plateau, across seeds, including at labels equal to context length.
The key ablation is the shifted-key one-layer model without the pointer. It has the same hardwired shift, so it differs from our model only in how the answer is emitted. If it matches the pointer model's width scaling, the pointer route contributes nothing and the core claim fails. A second ablation adds the pointer to the two-layer model. It tells us whether pointing is valuable on its own terms or only as a substitute for depth.
Why not just…?
Why not just use a two-layer Transformer, which learns induction heads without any gate? It does, eventually, and at large width the pointer model may well have no advantage. The cited evidence says the cost shows up as slow, seed-sensitive training and as failure when the data lacks a stepping stone, as in the label-equals-context regime of The mechanistic basis of data dependence and abrupt learning. Our width sweep and that regime are where the comparison is decided. If the two-layer model matches us at small width and has no plateau, we report that the pointer is unnecessary.
Why not just hardwire the previous-token shift and keep the normal output layer? This is the most serious threat, because the shift alone already makes one-layer induction representable. Our answer is the decoding-bottleneck hypothesis drawn from the $\sqrt{V}$ learnability bound in Learning to Recall with Transformers Beyond Orthogonal Embeddings, but that bound concerns specific training dynamics and might not bind in practice. The no-pointer shifted-key ablation is exactly this alternative. If its minimal width does not grow with $V$, we drop the central claim and keep only the gate and curriculum findings.
Why not just skip the curriculum and train on uniformly mixed data? On pure copying tasks we expect the curriculum to be irrelevant, and we say so. It matters only when the gate has to decide between copying and generating. Simple and Effective Curriculum Pointer-Generator Networks found the answerability ordering crucial in that setting, and Don't Copy That! documents what goes wrong without control. The uniform-mixing arm of the gate experiment tests this directly. A null result there would be reported as such.
Proposed Experiments
The first experiment is synthetic associative recall in the style of the induction-head analyses. Sequences contain random token pairs that recur. We sweep $V$ from hundreds to tens of thousands and $N$ from 64 to 2,048, with width from 8 to 512. The decisive check is the fitted slope of minimal successful width against $\log V$ and $\log N$ for each model. The prediction is a near-zero $V$-slope for the pointer model and a clearly positive one for the no-pointer shifted-key model.
The second experiment reproduces the in-context classification task with labels set equal to context length. The decisive check is whether the pointer model escapes the trap that catches the two-layer model, measured as the fraction of five seeds converging and the number of steps spent on a plateau.
The third experiment is the mixed copy-and-bigram task. We measure accuracy, gate calibration (the gap between the average value of $1-p_{gen}$ and the actual fraction of copyable targets), and retention of copy accuracy after the schedule ends, under curriculum versus uniform mixing.
Finally, as a sanity check outside synthetic data, we train small models on Tiny Shakespeare and Wikitext-2. We report perplexity on tokens whose answer appears earlier in context, separately from all other tokens, and compare the one-layer pointer model against the two-layer baseline at equal parameter count. The prediction is that the pointer model closes most of the gap on repeated tokens and none of it on the rest. That result would locate exactly what one layer plus a pointer buys and what still requires depth.
References
- Learning to Recall with Transformers Beyond Orthogonal Embeddings
- How Transformers Implement Induction Heads: Approximation and Optimization Analysis
- The mechanistic basis of data dependence and abrupt learning in an in-context classification task
- Separations in the Representational Capabilities of Transformers and Recurrent Architectures
- Representational Strengths and Limitations of Transformers
- A Copy-Augmented Generative Model for Open-Domain Question Answering
- LargePiG: Your Large Language Model is Secretly a Pointer Generator
- Simple and Effective Curriculum Pointer-Generator Networks for Reading Comprehension over Long Narratives
- Pointer-Generator Networks for Low-Resource Machine Translation: Don’t Copy That!
- Induction Heads Interpolate N-Grams
- Rethinking Associative Memory Mechanism in Induction Head