When a language model agent calls several tools in sequence, say looking up a company's current CEO and then booking a meeting with that person, it has two sources for every fact it uses. One is the tool's return value. The other is what the model memorized during training, which may be out of date. Existing studies of this "tool versus memory" conflict look at one tool call at a time and find that models swing between extremes. They abandon a correct memory for a wrong tool output over 80% of the time (MemToC), yet they fall back on outdated memory when retrieval is thin (RealTime QA). Nobody has measured what happens when the conflicting fact is not the final answer but an intermediate value that must be carried into the next call's arguments.
We propose to measure exactly where stale memory enters a multi-step tool chain. We think there are three distinct points. The model can skip the lookup and use its memory directly. It can call the tool and then type its remembered value, not the returned one, into the next call. Or it can pass the right value forward and still answer from memory at the end. We will build a procedurally generated test bed of tool workflows over real entities whose facts have changed, with tool names and graph shapes resampled per instance so that memorized benchmark patterns do not help. We will use it to show which entry point dominates and how leaks compound with chain depth. We will then test interventions matched to each entry point. The practical payoff is knowing whether the fix is a formatting change (bind arguments by reference), a planning change (force lookups for volatile facts), or neither.
Core Claim
Single-step conflict benchmarks cannot see two of the three failure points, and we expect those two to dominate in realistic chains. Concretely, we hypothesize the following. Skipped lookups grow when the model is confident about a fact. Argument overwriting grows with the distance in steps between where a value is produced and where it is consumed. Plain reference binding, where the plan cites an upstream output by identifier instead of retyping it, removes overwriting but does nothing for skipped lookups. Binding also amplifies the harm when the upstream tool is itself wrong. If binding alone closes most of the gap, the honest conclusion is that this is a plan-format problem rather than a knowledge problem, and the experiments are designed to report that outcome if it occurs.
How This Differs From Closest Work
Two papers motivate the gap. Investigating Tool-Memory Conflicts in Tool-Augmented LLMs measures conflict prevalence by contrasting memory-only and tool-only answers on static benchmarks such as MMLU and GSM8K, finding conflicts in 49.8% of instances on average. MemToC crosses memory correctness with tool correctness in a 2x2 design over 542 ToolHop facts. Both stop at one call. We reuse MemToC's 2x2 design and the memory-only probing from the first paper, but place the conflicting fact at varying depths inside a dependency graph. We score not only the final answer but every downstream argument.
Three papers supply method. DyVal generates reasoning problems as random directed acyclic graphs (DAGs: graphs in which each step depends only on earlier steps) to defeat test-set memorization, but has no tools or retrieval. We borrow its generator idea and its control over depth, width, and description order. ToolSandbox shows that scrambling API names exposes reliance on memorized tool conventions; we scramble every instance. OrchDAG represents multi-turn tool plans as DAGs and scores them by graph edit distance; we use the same plan representation so their trained models can be run directly.
Two papers constrain the design. Can Agent Memory Systems Track Evolving State? finds that drift toward stale, salient facts persists even under perfect retrieval and that chain-of-thought can worsen it. That result warns us not to assume that putting fresh values in context is enough, and it supplies our strongest structured baseline. Mitigating Temporal Misalignment by Discarding Outdated Facts predicts how long a fact typically stays true and triggers retrieval only for likely-expired facts. We reuse that idea as a gate for forced lookups, extended from single-hop question answering to planning decisions inside a chain.
ToolQA already isolates tool skill from memorization by using post-cutoff and private data. However, it deliberately removes the conflict: the model has no stale memory of private Airbnb data to fight. Our design requires the model to hold a confident, outdated belief, because that belief is the object of study.
Research Program
The generator. Each instance starts from a real entity-attribute pair with a time-varying value, such as an officeholder, a headquarters city, or a product's current version. We keep only pairs where a given model, probed closed-book with paraphrase checks, gives a stable answer. That answer is the model's memory value. The environment's current value is set in one of two ways. It may be a real post-cutoff change, or it may be a plausible synthetic edit drawn from the same type. Crossing this with whether the tool is correct yields MemToC's four cells. The edit is sampled per instance, so the environment's answer appears nowhere on the web and cannot be memorized.
Around this fact, the generator samples a DAG of two to six tools. In each graph the conflicting value is produced at a chosen node and consumed as an argument at one or more later nodes, at controllable distances. Tool names, parameter names, and description order are resampled per instance in the style of DyVal and ToolSandbox. Tools execute against a deterministic stateful backend, so every argument can be checked against ground truth.
Attributing each failure. Logging every call lets us assign each failure on a conflict item to exactly one entry point. A skipped lookup means the producing tool was never called and the memory value appears downstream. Argument overwriting means the tool was called, returned the environment value, and the memory value appears in a later argument. Final-answer reversion means all arguments were correct but the answer states the memory value. A small held-out set, about 300 traces, will be labeled by hand to check the automatic attribution.
A simple model of compounding. Let $\ell$ be the per-hop probability that the inherited value is replaced by memory, and $k$ the number of hops between producer and final consumer. If hops leak independently, end-to-end correctness of the inherited value would be
$$ P(\text{correct}) \approx (1-\ell)^k , $$
since each hop must independently preserve the value. We fit $\ell$ at $k=1$ and test whether deeper chains fall below this curve. Falling below would mean leaks are correlated, for example because the model "corrects" every mention once it doubts one. Staying above it would mean early correct use protects later use. Either result tells us something that single-step studies cannot.
The intervention. The method we propose has two parts, each aimed at one entry point. Argument overwriting is targeted by typed reference binding. The planner must fill a downstream argument with a pointer to an upstream output, and the executor substitutes the literal value; the model never retypes it. Skipped lookups are targeted by a volatility gate. A fact-duration predictor in the style of Zhang and Choi scores each fact the plan uses from memory. If the fact's typical lifespan is short relative to the time since the model's training cutoff, the plan is rewritten to insert a lookup. When an upstream tool output disagrees with a confident memory, the agent re-queries the tool or a second source instead of silently picking one. This targets the wrong-tool cells, where binding alone would propagate the error.
Baselines
The naive reference point is a ReAct-style agent: a loop in which the model alternately reasons in text and emits tool calls, with no special instructions. It is run on every model, including open 7–9B instruction-tuned models matching MemToC's set (Qwen2.5-7B, Llama, Mistral) and at least one frontier API model. All systems share the same generated instances, tool backend, and per-model memory probes.
The prompting baseline adds the vigilant and opinion-based prompts from Investigating Tool-Memory Conflicts, which reported conflict reductions of only 1–4 percentage points in single-step settings. We tune its wording on a development split with a budget equal to the one used for our gate threshold. Beating it means a larger reduction in total leak rate, with no loss in the wrong-tool cells.
The strongest existing structured competitor is StateMemWrapper from Can Agent Memory Systems Track Evolving State?. It enforces chronological value chains before answering and reportedly lifted StateMemBench accuracy by +32 to +67 points across six memory backends. It is the strongest competitor because it already rechecks dependent facts when a base fact changes, which is our problem moved into conversational memory. We adapt it by treating each tool return as a state update. No reported result found on Lacuna for StateMem on tool DAGs.
The strongest trained orchestrator is the OrchDAG GRPO-tuned Qwen2.5-7B. GRPO, group relative policy optimization, is a reinforcement learning method that scores each sampled plan against others for the same prompt. That model reached 40.2% accuracy on the OrchDAG test set and 47.1% solvable pass rate on StableToolBench, against 15–24% for zero-shot GPT-4o and Claude 3.5 Sonnet (OrchDAG). We run it as released and apply our intervention on top of it, so any gain is not just a matter of a better planner.
Three ablations isolate the ingredients: binding without the gate, the gate without binding, and the gate with forced lookups but no cross-checking. Beating the strongest baseline means a lower leak rate in the two tool-correct cells, at equal or better retention in the memory-correct/tool-wrong cell. It also means abstention F1 on ICT does not drop by more than a few points. That last check guards against the abstention leak MemToC observed, where fine-tuning Mistral to resist tools raised retention by 41.8 points but cut abstention by 57.7.
Why not just…?
Why not just tell the model to always trust tool outputs? In the tool-correct cells this may reduce argument overwriting. But it cannot fix skipped lookups, because a model that never calls the tool has nothing to trust. It also makes the memory-correct/tool-wrong cell worse, which is already the weakest at 6.5–17.1% retention in MemToC. The prompting baseline will show whether this holds in chains, and the per-cell reporting will expose any shift of errors from one cell to another.
Why not just bind arguments by reference, as code-style planners do? This is the real threat, and it is why binding appears both as an ablation and as half of our method. If binding alone closes nearly all leaks in the tool-correct cells and skipped lookups turn out to be rare, the knowledge-obsolescence framing adds little. We will report that. Our prediction is that skipped lookups rise sharply for confidently known facts and that binding faithfully propagates wrong tool outputs. The attribution step and the wrong-tool cells will settle the question.
Why not just force a lookup for every fact and never recall anything? This removes skipped lookups by construction. However, it multiplies calls, a serious cost: ComplexMCP reports single tasks consuming over 360,000 input tokens. It also offers no defense when the retrieved value is wrong; RealTime QA attributes 34% of errors to retrieval failures. We run always-lookup as an upper bound on skip prevention and compare the gate against it on leak rate per additional tool call.
Why not just use a long-context frontier reasoning model? StateMem's finding that drift persists under perfect retrieval argues against this, as does the observation that reasoning traces can rationalize stale facts. Still, the frontier model run on our generator tests it directly. If its leak rate is already near zero at every depth, the problem belongs to small models, and that finding narrows the claim.
Decisive Checks
The first check validates the generator. At depth one it should reproduce MemToC-scale harmful deference on the same open models. If it does not, the generated conflicts do not behave like real ones. The second check compares synthetic edits with real post-cutoff changes for the same entity types. They should produce matching leak rates per entry point within confidence intervals; a large gap would mean synthetic edits are a poor stand-in for real obsolescence. The third check is the depth curve against $(1-\ell)^k$, the result most clearly out of reach for single-step work. The fourth check tests for contamination. Performance on our resampled instances should track performance on the static ToolQA hard split, where ReAct with GPT-3.5 reached 8.2%, only for models that have not been heavily tuned on public tool benchmarks. A model strong on static benchmarks but weak on ours is evidence of pattern memorization. The intervention is worth adopting only if it beats binding alone on skipped lookups and wrong-tool retention without degrading ICT abstention.
References
- RealTime QA: What's the Answer Right Now?
- Investigating Tool-Memory Conflicts in Tool-Augmented LLMs
- DyVal: Dynamic Evaluation of Large Language Models for Reasoning Tasks
- ToolSandbox: A Stateful, Conversational, Interactive Evaluation Benchmark for LLM Tool Use Capabilities
- OrchDAG: Complex Tool Orchestration in Multi-Turn Interactions with Plan DAGs
- Can Agent Memory Systems Track Evolving State?
- Mitigating Temporal Misalignment by Discarding Outdated Facts
- ToolQA: A Dataset for LLM Question Answering with External Tools
- ComplexMCP: Evaluation of LLM Agents in Dynamic, Interdependent, and Large-Scale Tool Sandbox