When a frozen language model learns a task from a few examples in its prompt, theory suggests it behaves as if it runs a few steps of gradient descent on those examples inside its forward pass. How fast such an implicit learner converges depends on how well-conditioned its inputs are. In relation extraction, the task of deciding how two entities in a sentence are related, every demonstration carries entity names such as "Albert Einstein" or "Princeton." These names are mostly unique to one example and mostly irrelevant to the relation. We propose that they act as nuisance features for the implicit learner. They add nearly orthogonal directions that make its problem ill-conditioned and invite it to fit labels through names rather than through the relational phrase. The proposal is to replace names with typed placeholders such as [PERSON_1] in the demonstrations only. The query sentence keeps its real names, so the model still has the specific cues it needs to avoid false positives.
The idea yields a cheap intervention and a falsifiable mechanistic claim. The intervention should reduce the large, unexplained swings in in-context accuracy caused by entity-name choice, and should do so without the precision collapse that full masking causes. The mechanistic claim is that how badly a given set of names hurts should track a measurable conditioning statistic of the demonstration representations. The design has a useful side effect for multi-stage extraction systems. Demonstrations are gold-annotated, so their entity types are known, and the query is never masked. Upstream entity-type errors therefore cannot leak into the masking step.
Core Claim
Prior work leaves two findings unconnected. Robustness of Named-Entity Replacements for In-Context Learning shows that swapping entity names in demonstrations moves accuracy by up to 8.0 points on bAbI. It finds that this sensitivity does not correlate with token frequency, length or perplexity, and attributes it to unspecified "internal attention dynamics." Separately, Bypassing the Exponential Dependency: Looped Transformers Efficiently Learn In-context by Multi-step Gradient Descent proves that a linear looped Transformer running $T$ implicit gradient steps has prediction error bounded by
$$\text{Error} \le |\alpha| \exp\!\left(-\frac{T}{2\kappa}\right),$$
where $\alpha$ is a problem-dependent constant and $\kappa$ is the condition number of the in-context input matrix.
Our claim joins them. Suppose each demonstration's representation contains a shared relational part plus a name-specific part that is nearly unique per example. The name-specific parts then spread variance over many weakly supported directions. That inflates $\kappa$ and lets the implicit update $\Delta W \propto \text{Error}\times\text{Input}^\top$ (the form derived in Transformers Learn In-Context by Gradient Descent) fit labels through names.
Typed placeholders reuse the same few tokens across demonstrations, which collapses those directions. Random-string replacement (mention randomization, a known variant in the entity-masking literature) keeps every name unique. It should therefore not help, even though it removes world knowledge just as thoroughly. This contrast is the sharpest prediction in the proposal. It separates our conditioning account from the competing account that masking helps simply by removing memorized facts.
The asymmetry between demonstrations and query is what makes this more than a rerun of entity blinding. EMBRE: Entity-aware Masking for Biomedical Relation Extraction masks entities during pretraining. On BioRED it nearly doubles recall (0.1151 to 0.2173), but precision falls from about 0.40 to 0.23. That result constrains our design: it says removing entity information everywhere floods the output with false positives. We mask only where names are noise to the implicit learner, which is the examples it learns from, and keep them where they are signal, which is the instance it must label.
Research Program
The work has three stages, ordered so that each can falsify the next.
Stage 1: synthetic test with a known learner. We train linear and softmax looped Transformers on in-context regression. Each input concatenates a "type/relation" block shared across examples with an "identity" block drawn fresh for each demonstration. We vary the identity dimensionality relative to the number of demonstrations, and whether the identity block is predictive of the label.
We measure the empirical $\kappa$ of the context matrix, convergence across loops, and how much of the implicit update lies along identity directions. We then compare three treatments of the identity block: shared placeholder, fresh random replacement, and leaving it intact. Theory predicts that placeholders help in proportion to the reduction in $\kappa$, that random replacement does nothing, and that masking hurts when identity is predictive. This stage tests whether the mechanism is even coherent once linear attention is replaced by softmax, which is a known gap in the theory.
Stage 2: frozen LLMs on real text. We use open models in the 7–12B range so hidden states are accessible. We reproduce the name-swap variance on bAbI and CLUTRR, then extend it to relation extraction on CoNLL04, ADE, ACE05, SciERC and BioRED, framed as classifying a candidate entity pair given $k$ demonstrations.
For each prompt we compute a conditioning proxy from middle-layer demonstration representations: the condition number and effective rank of the demonstration hidden-state matrix after centering. We then ask two questions. Does this proxy predict accuracy across name substitutions, where frequency and perplexity did not? And does asymmetric typed masking both raise the proxy's conditioning and shrink the variance? We are explicit that a softmax model's hidden states are only a proxy for the theory's input matrix. A null correlation here would be informative, not something to explain away.
Stage 3: pipeline and cost. End-to-end extraction feeds candidate spans from an entity detector into pairwise relation scoring. With $m$ detected entities there are $m(m-1)$ ordered pairs to score, which is the pairwise blow-up that span-based systems such as End-to-end Neural Coreference Resolution contain by pruning. Because the masked demonstration block is identical for every pair, we encode it once as a shared cache and score each pair with only the query tokens. This keeps per-pair cost independent of the number of demonstrations.
We then inject controlled noise into upstream entity boundaries and types and measure how relation F1 degrades. Asymmetric masking should be insensitive to type noise, since query types are never used. Symmetric typed masking, which needs predicted types on the query, should inherit the upstream error. This tests pipeline error propagation directly rather than just citing it.
Baselines
The naive reference point is standard few-shot prompting with original entity names. Demonstrations are sampled identically across all conditions, with the same $k$, the same model and the same pair-classification format. We report its mean and its spread across ten name resamplings, because the spread is a primary outcome.
The strongest existing method in the same frozen-model regime is Context Tuning with key-value optimization, from Context Tuning for In-Context Optimization. It directly optimizes the cached demonstration states with leave-one-out masking and token dropout. It reports a +6.1% gain over standard prompting with Mistral-12B on NLP-LR, MMLU, BBH and ARC, while running roughly 2x faster than test-time training. We found no reported result on Lacuna for it on relation extraction. We will run it on unmasked and on masked demonstrations, with the same optimization step budget and the same demonstrations. Beating it means asymmetric masking without any optimization matches or exceeds its F1 at lower cost. Alternatively, masking added to it gives a gain beyond its confidence interval across seeds and name resamplings.
The test-time ensembling baseline is majority voting over ten name resamplings, which the name-replacement paper reports gives a 1.6-point gain on CLUTRR. We run it at its natural tenfold cost and compare at matched accuracy and at matched compute.
Fully supervised extractors are reported as a ceiling, not a competitor. These are ITER from ITER: Iterative Transformer-based Entity Recognition and Relation Extraction (71.9 relation F1 on ACE05, 85.6 on ADE), HGERE from Joint Entity and Relation Extraction with Span Pruning and Hypergraph Neural Networks (55.7 relation F1 on SciERC, 67.5 on ACE05), and CMAN from Modeling Dense Cross-Modal Interactions for Joint Entity-Relation Extraction (72.97 relation F1 on CoNLL04). We also retrain ITER on the same $k$ labeled examples the prompt sees, which is the fair low-resource comparison.
The ablations that isolate the key ingredient form a two-by-two design: masking on demonstrations or not, crossed with masking on the query or not. This design is aimed at the precision question EMBRE raises. We cross it with placeholder type: typed, generic [ENTITY], and unique random strings. The random-string arm is the critical control for the conditioning claim.
Why not just…?
Why not just fine-tune a small extractor like ITER? With full training data it will win on F1; the ceiling numbers above say so, and we do not claim otherwise. The real threat is the low-resource regime, where ITER trained on the same 16–64 examples may match frozen-model prompting. We run exactly that comparison. If it wins, the practical claim shrinks to settings where retraining is unavailable, and the contribution rests on the mechanistic finding.
Why not just average over many name choices? Ensembling does reduce name variance, but the reported gain is modest (1.6 points on CLUTRR) and costs ten forward passes per query. Multiplied across $m(m-1)$ candidate pairs per document, that cost is prohibitive. If masking matches the ensemble at one-tenth the cost, masking is preferred. If the ensemble is clearly better, the conditioning account is incomplete.
Why not just let Context Tuning absorb the name noise? This is a genuine threat. Optimizing the cache could learn to ignore name directions on its own. We test it on unmasked versus masked demonstrations. If optimization alone closes the gap, the practical recommendation becomes "optimize the cache." The conditioning account would then still predict that masked caches converge in fewer optimization steps, and we measure that.
Why not just mask everything, as entity blinding does? EMBRE's BioRED result shows the cost: precision dropped from about 0.40 to 0.23. Symmetric masking also makes the query depend on predicted entity types, so upstream type errors cascade into relation decisions. The two-by-two ablation and the type-noise injection measure both effects directly.
Decisive Checks
Three outcomes would settle the proposal.
First, in Stage 1, typed placeholders must lower $\kappa$ and speed convergence while fresh random strings do not, and this must hold under softmax attention as well as linear. If random strings help equally, the mechanism is removal of knowledge rather than conditioning, and we report it as such.
Second, in Stage 2, the hidden-state conditioning proxy must explain a meaningful share of name-swap variance on bAbI and CLUTRR, where frequency and perplexity were reported not to. Asymmetric masking must also cut the across-name standard deviation in relation F1 on CoNLL04, ADE, ACE05, SciERC and BioRED.
Third, in Stage 3, asymmetric masking must hold precision near the unmasked level while recovering recall, the opposite of EMBRE's trade-off. Its F1 must also stay flat under injected entity-type noise while symmetric masking degrades.
If the first two checks pass and the third fails, we have explained the name sensitivity of in-context learning but not fixed extraction. That is still a publishable negative result, and it would point toward query-side treatments instead.
References
- Robustness of Named-Entity Replacements for In-Context Learning
- Bypassing the Exponential Dependency: Looped Transformers Efficiently Learn In-context by Multi-step Gradient Descent
- Transformers Learn In-Context by Gradient Descent
- EMBRE: Entity-aware Masking for Biomedical Relation Extraction
- End-to-end Neural Coreference Resolution
- Context Tuning for In-Context Optimization
- ITER: Iterative Transformer-based Entity Recognition and Relation Extraction
- Joint Entity and Relation Extraction with Span Pruning and Hypergraph Neural Networks
- Modeling Dense Cross-Modal Interactions for Joint Entity-Relation Extraction