Surfacing Disagreement Beats Resolving It: A Gated Ablation Study of Provenance-Weighted Memory for Multi-Agent Debate

Ziaullah Khan, Abdul Rehman Khalid, Shrestha Nischal Lal, Nheng Vanchhay, Md Ariful Islam Mozumder, Hee-Cheol KimarXiv 2026 · arXiv

TL;DR

When multi-agent systems share a memory, the standard "design reflex" is to resolve conflicting facts before they reach the agents. This paper demonstrates that surfacing these contradictions—letting the agents see the disagreement and its provenance—dramatically improves accuracy and robustness against adversarial attacks compared to silently picking a "winner" at the storage layer.

What Problem Does This Paper Address?

In multi-agent debate systems, agents often rely on a shared memory (like a knowledge graph or vector database). A common assumption in the field is that this memory should be consistent. If the system retrieves two conflicting facts (e.g., "The CEO is Alice" vs. "The CEO is Bob"), the storage layer usually employs a heuristic—like choosing the most recent entry or the one with the highest confidence score—to provide a single "clean" answer to the agents.

The authors argue this is a "strange division of labor." A debate system is specifically designed to adjudicate competing claims through reasoning. By resolving conflicts at the storage layer, we use the system's weakest tool (a simple heuristic) to bypass its strongest tool (the reasoning agents). Furthermore, this "silent reconciliation" makes the system brittle; an adversary who knows the tie-breaking rule (like recency) can easily "poison" the memory.

Methods

The authors built DebateGraph, a modular memory service where various mechanisms from current literature can be toggled as independent "switches."

Figure 1: DebateGraph, left to right. Conversations and documents are ingested with planted contradictions mixed in; extraction produces provenance-annotated triples; the write trust-gate admits, rejects or supersedes them into a bi-temporal store shared by all backends. At query time a router retrieves under a hard token budget, and the resolver groups retrieved edges that assert different objects for the same relation into ranked conflict sets. The one switch this paper is about sits at the end of that stage: COLLAPSE passes only the top-ranked claim, SURFACE passes the whole ranked set with its provenance. Everything downstream, the three agents, the two rounds and the judge, is identical in both arms. Each evaluated mechanism is one independent switch, so a single ablation changes exactly one thing.

The core experiment compares two policies:

  • COLLAPSE (The Baseline): The memory layer ranks conflicting claims and only returns the top-ranked "winner" to the agents.
  • SURFACE (The Proposal): The memory layer returns the entire ranked set of conflicting claims, along with their provenance (who wrote it, when, and from what source).

The study used a "gated ablation" protocol: they pre-declared performance thresholds for five different mechanisms (like write-time trust gates and retrieval routers). If a mechanism didn't meet its threshold in controlled runs, it was considered "refuted" or "failed."

Datasets

The researchers evaluated their system using:

  • LoCoMo: A dataset of multi-turn conversations.
  • HotpotQA: Used for cross-corpus replication to ensure the findings weren't limited to conversational data.
  • DebateMem-Conflict: A new, deterministic "poisoning" set introduced by the authors. It contains 122 items where a "lie" is planted that contradicts a gold-standard fact. The lie is attributed to a non-debating writer and given a more recent timestamp to trick recency-based heuristics.

Key Insights and Ideas

  • The "Storage Layer" shouldn't be a Judge: Surfacing conflict helps because agents can use the provided provenance to discount unreliable sources or identify contradictions that require deeper reasoning.
  • Groundedness is a Trap: One of the "failed" mechanisms was a write-time trust gate designed to filter out "un-grounded" claims. It failed because adversaries are smart: they compose lies using entities already present in the graph. To a simple algorithm, a lie about a known entity looks more "grounded" than a truthful first mention of a new entity.
  • The Retrieval Bottleneck: Initially, the benefit of surfacing contradictions seemed small. The authors discovered a "harness bottleneck": their retrieval cap was too low. If the memory doesn't retrieve the contradiction in the first place, the agents can't debate it. Raising the retrieval cap (from 40 to 200 edges) multiplied the positive effect of surfacing by a factor of eight.
  • Some "Features" are Bugs: Mechanisms like "router provenance features" actually harmed accuracy once contradictions reached the prompt. Adding complexity to the retrieval logic often introduced more noise than signal.

Key Results

The results were decisive in favor of transparency over silent reconciliation:

  • Accuracy Boost: On the graph backend, surfacing contradictions improved the F1 score by +0.340 (a massive jump from the initial +0.043 measured before fixing the retrieval bottleneck).
  • Adversarial Robustness: The "SURFACE" approach cut the success rate of adversarial attacks from 22 down to 7 (out of 122 poisoned items).
  • Cost Efficiency: These gains were achieved at "equal cost," meaning the token budget for the agents remained the same; the system simply used those tokens to show the conflict rather than hiding it.
Figure 2: Accuracy–cost frontier over the 37 LoCoMo runs (the two HotpotQA runs are excluded as a different corpus). The x-axis is broken: the two regimes are an order of magnitude apart in cost, so each is drawn at its own scale and the empty middle is the point rather than an artefact. No configuration is simultaneously cheap and accurate, and the λ sweep collapses into a single cluster rather than tracing a trade-off. The exception is p3_surface_reach at F1 0.456 for 11.5K tokens (Table 2), which lifts the retrieval ceiling by 0.07 over every other retrieval run at no extra cost. Only runs discussed in the text are labelled.

As shown in Figure 2, the p3_surface_reach configuration (surfacing with a sufficient retrieval cap) significantly lifted the retrieval ceiling compared to all other configurations.

Limitations and Future Work

  • Model Specificity: All tests were performed using the gpt-oss-120b family. While the authors attempted to replicate results with a 70B model, the inference speed was too slow for the full 39-run matrix.
  • Context Window Limits: In these experiments, the total memory never actually exceeded the model's 32K context window. The authors note that in a regime where memory genuinely overflows, some form of reconciliation might become a physical necessity rather than a design choice.
  • Adversarial Sophistication: The planted contradictions were template-generated. Future work could explore how agents handle more "fluent" or subtle adversarial paraphrases.

The paper concludes with a strong recommendation for engineers: Preserve the disagreement; let the reasoner reason. Silently reconciling memory is an assumption worth abandoning.

Summary updated

Abstract

Shared memory for multi-agent debate is usually built to resolve inconsistency: retrieval returns one winning fact and the disagreement disappears before any agent sees it. We ask whether that is correct. In DebateGraph, four backends share one contract (read/write/snapshot), so five mechanisms from the agent-memory literature can be removed independently under a fixed budget, each gated on a pre-declared threshold. Across 39 runs one survives: preserving contradictions and passing the ranked conflict set to the debate, rather than collapsing it to a winner, improves F1 by +0.340 (95% CI [+0.267, +0.414]) and cuts attack success from 22 to 7 of 122 poisoned items (McNemar exact p=0.0003), at equal cost. Reaching it meant fixing a bottleneck in our harness: conflict sets are built only over retrieved edges, and our pre-declared cap left three quarters of planted contradictions unretrievable, first measuring the mechanism at +0.084. The effect replicates on two retrieval substrates and directionally on HotpotQA documents. No other mechanism survives, and two are worse than inert: once contradictions reach the prompt, a write trust-gate and the router's provenance features each cost accuracy (-0.075 and -0.096 F1). The gate fails structurally: poisoned objects are always entities the graph already contains, against 53% of gold, so groundedness scores the lie above a truthful first mention. We release the harness, attack set, and logs, and argue that silently reconciling memory is the assumption worth abandoning.
BibTeX
@misc{khan2026surfacing,
  author = {Ziaullah Khan and Abdul Rehman Khalid and Shrestha Nischal Lal and Nheng Vanchhay and Md Ariful Islam Mozumder and Hee-Cheol Kim},
  title = {{Surfacing Disagreement Beats Resolving It: A Gated Ablation Study of Provenance-Weighted Memory for Multi-Agent Debate}},
  year = {2026},
  eprint = {2611.22461},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.22461},
}
Figure