Adversarial Training Aimed at the Pointer Heads

Language models keep track of which value belongs to which variable when they read code, follow kinship stories, or carry quantities through a math problem. Recent interpretability work shows this "variable binding" (keeping a role such as x or the aunt attached to the right value) is carried out by a small set of attention heads. These heads act like pointers: they read from and write to positions in the model's running internal state. They are also fragile. Shortcut circuits learned early in training are never removed, only overridden. Worst-case inputs push circuits far from their average-case behavior. Adversarial training, which trains the model on deliberately constructed worst-case inputs, is the standard fix, but it multiplies training cost by the number of attack steps in its inner loop.

This note proposes running both the attack and the defense on the binding pointers alone. The adversary perturbs only the outputs of the heads that implement binding, at the layers where they sit. The defender updates only those heads. If the brittleness of binding really is concentrated in these heads, this should recover most of the robustness of full latent adversarial training at a small fraction of the cost. The method should also preserve the systematic dereferencing circuit instead of collapsing it back to shortcuts. If the brittleness is not concentrated there, a matched random-head control will show it, and we will report that.

Core Claim

The prior work points in one direction. Two papers show the binding computation is sparse and identifiable. Discovering Variable Binding Circuitry with Desiderata isolates 10 of 1,640 components in LLaMA-13B (9 attention heads, 1 MLP); patching them gives 84% value dependence and 82% operation invariance. How Do Transformers Learn Variable Binding in Symbolic Programs? finds dedicated read/write routing heads in a small transformer trained on assignment chains.

Two other papers motivate the worry that this sparse machinery breaks under attack. Adversarial Circuit Evaluation shows the entity-tracking circuit for indirect object identification reaches a maximum KL divergence of 14.64 from the full model on adversarial inputs. Improving the Robustness of Transformer-based Large Language Models with Dynamic Attention attributes text attacks to "attention hijacking," where malicious tokens pull attention onto irrelevant positions.

The cost side supplies the method we build on. Efficient LLM Adversarial Training via Low-Rank Defense and Circuit-Guided Surrogates (LAT-ReFT) cuts latent adversarial training cost by pruning MLP neurons for a surrogate attacker and by training low-rank interventions on a token window. It never touches attention routing and is evaluated only on classification and password matching.

Our claim is specific. On binding tasks, an adversary restricted to perturbing the outputs of causally identified binding heads finds failures that transfer to real input-level attacks. A defense that updates only those heads against this adversary closes most of the gap to full latent adversarial training on dereferencing tasks. It also costs fewer FLOPs than LAT-ReFT, because the inner loop only runs from the first circuit layer upward.

Formally, let $h_c(x;\theta_c)$ be the concatenated outputs of the binding heads $c$ (parameters $\theta_c$, all other weights frozen) and $\mathcal{L}$ the task loss on answer $y$. We solve

$$\min_{\theta_c} \; \mathbb{E}_{(x,y)} \max_{\|\delta\|\le\epsilon} \mathcal{L}\big(f(x;\, h_c(x;\theta_c)+\delta),\, y\big).$$

The activations below the earliest circuit layer $l$ are computed once per batch and cached. Each inner step therefore costs roughly a fraction $(L-l)/L$ of a full forward-backward pass, where $L$ is the number of layers. The outer gradient touches only $\theta_c$.

Research Program

Two settings. The first is a controlled one. We replicate the 37.8M-parameter GPT-2-style model trained on the synthetic 17-line program dereferencing data from Wu et al. In that setting the routing circuit and the leftover line-position shortcut are both known, and length generalization to 13 hops is a clean test of whether systematic binding survives. The second is Llama-3.1-8B, chosen to match the LAT-ReFT baseline. We rediscover its binding circuit with the value-dependence and operation-invariance mask optimization of Davies et al. on their arithmetic binding templates. The cost of this discovery is counted against our method.

Restricting the defense. We train either the full parameters of the binding heads or a low-rank adapter on their query, key and value projections, matched in parameter count to LAT-ReFT.

Circuit drift. Adversarial training may move the computation out of the heads we defend. We therefore re-run circuit discovery every fixed number of steps and report how stable the identified set is.

Diagnostics. After training, we measure three things. First, worst-case circuit faithfulness using the resample-ablation KL protocol of Adversarial Circuit Evaluation. Second, value dependence and operation invariance of the circuit. Third, whether the model has regressed to the line-1 shortcut that Wu et al. found suppressed but intact.

Baselines

Undefended model. The naive reference point is the model after ordinary task fine-tuning. It gives clean accuracy and the attack success rate that every defense must lower. On the program task, Wu et al. report above 99.9% clean test accuracy. No reported adversarial result on this task was found on Lacuna, so we establish it ourselves.

Full latent adversarial training. This is the robustness ceiling and the cost ceiling. It applies inner-loop perturbations to the residual stream at the same layers and updates all weights. On Llama-3.1-8B, LAT-ReFT reports that full LAT drives GCG attack success rate to 0.02 on its classification tasks. This is a classification-task number and not a binding result.

LAT-ReFT. This is the strongest efficient competitor because it attacks the same two costs we do. We run its released recipe on our tasks with the same data, the same attack families, the same number of inner steps and the same wall-clock and FLOP budget, tuning its learning rate and intervention rank with the same number of trials we give our method. LAT-ReFT reports GCG attack success rate falling from 0.97 to 0.15 on Llama-3.1-8B, with 48.1% fewer training FLOPs than full LAT. No reported result on Lacuna applies it to variable binding tasks. Our method beats it if, at equal or lower FLOPs, it achieves lower attack success on binding tasks with clean accuracy and 13-hop generalization within one point.

ProTransformer. ProTransformer replaces attention's weighted average with a robust estimator at inference time. It needs no training but reports 1–2x inference latency. We include it as the strongest training-free option.

Data augmentation. Supervised fine-tuning on discrete adversarial binding examples, in the style of Explore Theory of Mind, is given the same compute budget.

Random heads (key ablation). This ablation isolates our central ingredient. It runs the identical algorithm on a random set of heads matched in count and layer distribution to the binding circuit. A second ablation perturbs the whole residual stream at those layers instead of only the circuit head outputs.

Why not just…?

Why not just run LAT-ReFT on binding tasks? This is the default, and it may work. Its own reported gap (0.15 against 0.02 for full LAT) suggests that low-rank interventions on a token window leave deeper routing exposed. Binding failures plausibly live in exactly that routing, since the binding circuits are attention heads and its surrogate prunes only MLP neurons. The equal-FLOP comparison above settles it directly. If LAT-ReFT matches us on dereferencing robustness, the circuit targeting adds nothing.

Why not just do latent adversarial training with an adapter on any mid-layer heads? This is the most serious threat. The benefit could come entirely from restricting the update to a few mid-to-late attention parameters, regardless of which ones. The random-heads and full-residual ablations exist to test this. The proposal's mechanistic claim holds only if circuit heads clearly beat matched random heads on held-out input attacks.

Why not just augment training data with adversarial distractor assignments? It has no inner loop and is cheap. The EXPLORETOM fine-tuning results (+27 points on ToMi) show data-level adversaries help. The open question is coverage. Discrete augmentation covers the attack types we enumerate, while a latent adversary on the pointer heads may cover routing failures we did not anticipate. We test on attack families held out from both methods' training.

Why not just use an inference-time defense like ProTransformer? It avoids training cost but pays 1–2x latency on every token. Robust reweighting of attention may also blunt the sharp, content-independent pointer lookups that dereferencing needs. Measuring its clean 13-hop accuracy will show whether that happens.

Proposed Experiments

Benchmarks. We use three: the 17-line program dereferencing task, the Davies et al. arithmetic binding templates, and CLUTRR. In CLUTRR, the best text model reaches 0.67 clean accuracy and drops to near random under injected noise. The facebook/ExploreToM stories serve as a transfer test for role tracking.

Attacks. Attacks used during training are GCG suffixes and inserted distractor assignments. Held-out attacks are variable renaming that exploits token residue, reordering of assignment lines, and CLUTRR-style noise sentences.

Metrics. We report accuracy and attack success rate under attack, clean accuracy, maximum hop count dereferenced, training FLOPs including circuit-discovery cost, and post-training worst-case circuit KL.

Decisive checks. The project succeeds if three things hold:

  1. Circuit-targeted training beats matched random heads on held-out input attacks.
  2. It matches or beats LAT-ReFT at equal or lower FLOPs.
  3. It does not revive the line-position shortcut.

A further check matters as much. The circuit-restricted adversary exploits the circuit's local brittleness, and Adversarial Circuit Evaluation shows circuits can be unfaithful in exactly the worst case. If input-level attacks still succeed after training, those attacks are bypassing the defended pointers. Detecting that bypass is the most important negative outcome to measure.