When a language model fails a specialist question, the usual fix is to hand it the missing knowledge. Text retrieval does this by pasting documents into the prompt. Retrieval-augmented architectures do it by letting the model attend to stored passages through dedicated layers. A third route uses a few worked examples. The puzzling result is that handing over the rule often does not help. On TheoremQA, prepending textual definitions of the needed theorem gave open models no significant gain, and they stayed below 15% (TheoremQA). Even when knowledge injection does seem to work, we usually cannot tell whether the model applied the supplied rule or recalled a benchmark it saw during training.
We propose to separate these explanations with one instrument: procedurally generated problems built on rules that are either new to the model or that deliberately contradict the textbook version. We then deliver those rules to a frozen model through four channels and measure, for each, how often it applies the supplied rule and how often it falls back on the version it memorized. The four channels are a written definition in the prompt, worked examples in the prompt, an internal "task vector" patched into the hidden states, and an external memory read through cross-attention. Because the rules are invented, contamination is impossible by construction. Because each counterfactual rule has a canonical twin, the gap between the two directly measures how much of the model's apparent domain skill is memory.
Core Claim
The claim is that the "knowledge-to-application" failure TheoremQA documented depends on the channel, and that a contamination-proof counterfactual benchmark can show which channel lets a frozen model apply a rule it has no parametric support for. We expect two concrete outcomes. First, worked examples should work where written definitions do not, whether they sit in the prompt or are compressed into an internal vector. Second, the advantage of attention-integrated memory should appear only when the needed rule must be found in a large library that cannot fit in the prompt.
Either outcome is informative. If prompt-level worked examples dominate at every library size, the case for architectural knowledge injection in expert domains weakens considerably, and we will report that plainly.
Grounding in Prior Work
Four lines of work frame the study.
The first motivates it. TheoremQA shows the gap and the failure of text augmentation, but it tested only prompt-level methods and never audited whether GPT-4's 51.0% under Program-of-Thought prompting reflects contamination.
The second supplies the injection mechanisms we reuse:
- RETRO splits input into chunks and lets each chunk attend, through chunked cross-attention, to retrieved neighbors encoded separately (RETRO).
- LONGMEM caches demonstrations as key-value memories read by a small trained side network on a frozen backbone, which scales in-context learning to thousands of examples (LONGMEM).
- AtlasKV compiles facts into attention-readable key-value pairs at billion scale (AtlasKV).
All three were evaluated on generic language understanding or single-hop factual QA, never on multi-step rule application. AtlasKV itself notes that flattening knowledge into triples hurts multi-hop reasoning. That is a constraint our depth-controlled problems test directly.
The third gives us the internal-vector channel. Hendel, Geva and colleagues showed that demonstrations compress into a mid-layer "task vector" that, patched into a query-only pass, recovers 80–90% of in-context accuracy (In-Context Learning Creates Task Vectors). Bu et al. give a theory in which in-context learning works as adding a retrieved concept vector to the query (Provable In-Context Vector Arithmetic). Both are limited to single-token mappings. Neither can say whether a task vector encodes a rule or a key into memorized data. Counterfactual rules are exactly the test that distinguishes these.
The fourth supplies the evaluation lens and also constrains naive designs:
- DyVal generates fresh reasoning problems from directed acyclic graphs (DAGs, here meaning dependency graphs of steps) and finds that depth, not width, drives failure (DyVal). It covers only arithmetic, logic and graph algorithms, not domain rules.
- Golchin et al. show that adding demonstrations surfaces memorized training data, with accuracy gains tracking memorization at a Pearson correlation above 0.90 (Memorization In In-Context Learning). This warns that any channel's gain on a real benchmark may be memory.
- CoDeC detects contamination because demonstrations lower a model's confidence on memorized items and raise it on unseen ones (Detecting Data Contamination via In-Context Learning). It has never been run with context delivered through cross-attention rather than the prompt.
Research Program
The benchmark. We will build a generator we call CounterRule. It starts from formula-shaped theorems drawn from TheoremQA's physics, electrical engineering and finance coverage, for example compound interest, kinematics and circuit laws. Each theorem gets three variants:
- Canonical: the textbook rule.
- Neutral-novel: an invented, named law with plausible units and no textbook counterpart, so the model has no prior about it.
- Conflicting: the textbook law with an altered exponent, coefficient or variable role, so the memorized version gives a wrong answer.
Following DyVal's construction, problems chain one to four rule applications into a dependency graph. They are rendered into natural language with fresh numbers on every draw and graded numerically.
The conflicting condition gives a sharp error signature. An answer that matches what the canonical rule would produce on the same numbers is a reversion, meaning the model ignored the supplied knowledge and used its memory. We define the memorization dividend of a model and channel as
$$\Delta = \text{acc}_{\text{canonical}} - \text{acc}_{\text{neutral}},$$
where both accuracies are measured on problems with identical graph structure and depth. Neutral-novel rules carry no prior, so $\Delta$ estimates how much canonical accuracy comes from recall rather than application.
The channels. Each rule is delivered to a frozen backbone in one of four ways:
- a written definition prepended to the prompt, which replicates TheoremQA's augmentation;
- three to eight worked examples prepended to the prompt;
- a task vector extracted from a worked-example pass and patched into the query pass, using the layer sweep of Hendel, Geva and colleagues;
- the same worked examples stored in a LONGMEM-style external memory read through a trained cross-attention side network.
The task-vector channel is exploratory. The original method was shown only for single-token outputs, and our answers span several tokens. We will patch at the position before generation begins and report honestly if the effect does not survive beyond the first token.
Library-size scaling. We vary the number of candidate rules the system must choose among, $L \in \{1, 10, 100, 1000\}$. At small $L$ every channel can hold the whole library. At large $L$ the prompt channels need a retriever, while the memory channel attends over everything.
Backbones. We use 7B–30B open models of the LLaMA family, as in the task-vector work, so that hidden states are accessible. Where retrieval-pretrained checkpoints of the kind compared in GPT vs RETRO are available, we add them as a second form of cross-attention injection.
Baselines
Naive reference. The model gets the query with no rule at all. On neutral-novel and conflicting rules it should score near zero. On canonical rules its score is pure parametric recall, so it anchors the memorization dividend.
Replication of TheoremQA. The written-definition channel reproduces TheoremQA's augmentation. TheoremQA reports open models below 15% and GPT-4 at 51.0% with Program-of-Thought prompting, and says only that augmentation brought no significant gain. No reported augmentation figure was found on Lacuna. We therefore re-run real TheoremQA ourselves under this channel so that our numbers on CounterRule and on TheoremQA are comparable.
Strongest existing method. The strongest competitor is prompt-level worked examples. It is the cheapest channel, and the task-vector work shows demonstrations carry most of the useful signal. It gets the same examples, the same number of tokens of rule material, and the same tuning budget for example count and order as every other channel. At large $L$ it is paired with BM25 retrieval of the top-k rule cards, so it competes with the memory channel on equal information.
Cross-attention memory. This channel counts as beating the prompt baseline only if two conditions hold together: it lowers the reversion rate on conflicting rules, and it raises neutral-novel accuracy at matched $L$ and depth. A win at $L=1$ is not required. A win at $L \ge 100$ with a matched retriever is the claim.
Ablation of the memory channel. We keep the side network but fill the memory with written definitions instead of worked examples. This separates "attention-delivered" from "example-shaped" knowledge.
Weight-update reference. LoRA fine-tuning on the same worked examples serves as an upper reference. It changes weights for each rule set, so it is not a fair competitor for retrieval at scale. It does tell us how much application is achievable at all.
Why not just…?
Why not just put worked examples in the prompt? This is the real threat. At small library sizes we expect it to match or beat every other channel, and the design lets it. What the literature does not settle is behavior once the right rule must be found among thousands. There, prompt retrieval errors and dilution of attention over long context compete against the memory channel's learned access. Our $L$ sweep settles this directly. If prompt examples with retrieval win at $L=1000$, we report that architectural injection buys nothing for rule application.
Why not just fine-tune with LoRA? For a fixed rule set LoRA may be best, which is why it is our upper reference. But it changes weights every time the library changes, and the GPT-vs-RETRO comparison shows its gains come from task-specific training rather than zero-shot access to new knowledge. Our neutral-novel condition with held-out rules measures exactly that. We will also report whether fine-tuning on conflicting rules causes canonical accuracy to drop, which would show the edit is overwriting memory rather than adding a rule.
Why not just use a bigger model? Scale may raise canonical accuracy while increasing reversion, since stronger priors resist counterfactual rules. The file has no direct evidence on this, so we test it. We run the same channels across 7B, 13B and 30B backbones and plot reversion rate against size.
Why not just use DyVal or audit TheoremQA with CoDeC? DyVal is contamination-proof but contains no domain knowledge. CoDeC flags contamination but cannot say whether an uncontaminated model can apply a supplied rule. We use both as components: DyVal's generator builds our problems, and CoDeC audits real TheoremQA for each backbone so we can check whether its contamination score predicts the measured memorization dividend $\Delta$.
Proposed Experiments and Decisive Checks
First check: is the TheoremQA failure about format? On canonical and neutral-novel CounterRule, compare written definitions against worked examples at $L=1$. If examples succeed where definitions fail on neutral rules, TheoremQA's negative result reflects the form in which knowledge was delivered, not missing knowledge.
Second check: which channel overrides memory? On conflicting rules, report reversion rate by channel and by depth from one to four steps. A channel that truly delivers knowledge should keep reversion low as depth grows. DyVal's depth decay predicts every channel will degrade, and the question is which degrades slowest. This is the first test of whether task vectors carry rules or memory keys. A task vector that recovers canonical accuracy but produces reversions on conflicting rules is acting as a retrieval key.
Third check: where does the memory channel win? The $L$ sweep gives the crossover point, if any, where cross-attention memory beats prompt examples with matched retrieval.
Fourth check: is the contamination signal masked? We run CoDeC with demonstrations delivered through the memory channel instead of the prompt, on corpora known to be seen versus unseen. We then test whether the confidence-drop signature survives. If it does not, attention-integrated models need their own contamination audits, which is a practically important finding for anyone evaluating retrieval-augmented systems.
All accuracies, reversion rates and memorization dividends will be reported with confidence intervals from repeated sampling of fresh generator instances.
References
- TheoremQA: A Theorem-driven Question Answering Dataset
- Improving Language Models by Retrieving from Trillions of Tokens
- Augmenting Language Models with Long-Term Memory
- AtlasKV: Augmenting LLMs with Billion-Scale Knowledge Graphs in 20GB VRAM
- In-Context Learning Creates Task Vectors
- Provable In-Context Vector Arithmetic via Retrieving Task Concepts
- DyVal: Dynamic Evaluation of Large Language Models for Reasoning Tasks
- Memorization in In-Context Learning
- Detecting Data Contamination in LLMs via In-Context Learning
- GPT vs RETRO: Exploring the Intersection of Retrieval and Parameter-Efficient Fine-Tuning