Peer-to-Peer Delphi Panels for Partitioned Private Knowledge

When useful knowledge is split across many machines that cannot pool their documents, such as hospitals, edge devices, or separate organizations, a language-model agent on one machine cannot simply retrieve what it needs. It can, however, ask the right peers what they conclude. We propose to treat each question as a small, temporary expert panel. A semantic overlay network supplies the panel. This is a peer-to-peer network whose links connect machines holding related content, so a query can be passed toward peers likely to know about its topic. The panel then deliberates by the Delphi method, a survey protocol developed for human expert panels. In Delphi, experts answer independently, see only an anonymous summary of the group's answers and reasons, are asked to justify themselves if they sit far from the group, and stop when the spread of answers falls below a preset threshold.

The claim is narrow and testable. Recent work shows that letting language-model agents talk over several rounds usually does not beat a simple vote over their independent first answers. That result is proven for agents that all see the same context. We argue that iterative exchange earns its cost only when the evidence is spread across agents who cannot see each other's documents, and especially when the answer requires combining facts that live on different peers. The Delphi design of anonymous summaries, justification demanded only from outliers, and a statistical stopping rule is the right way to carry that exchange without the conformity collapse that free-form agent debate suffers.

What exists and where the gap is

Two papers sit on either side of this idea. Distributed Retrieval-Augmented Generation builds DRAG, in which edge nodes keep private knowledge bases. A Topic-Aware Random Walk (TARW) routes each query toward peers that have answered well on its topic in the past, and the snippets that come back are used for generation at the asking node. DRAG reports an F1 of 83.92% on MMLU against 85.75% for centralized retrieval, with roughly half the messages of flooding. That paper supplies our routing method. Its model is still one-way snippet fetching: there is no exchange of judgments and no way to reconcile contradictory peers. It also depends on snippets being shareable, which is exactly what strict privacy settings forbid.

An LLM-based Delphi Study to Predict GenAI Evolution automates Delphi with persona-prompted GPT-4o-mini agents and a single central Organizing Agent. Over five rounds it produces thematically stable output, and the authors name the lack of retrieval as the key weakness. That paper supplies the protocol template. We differ from it on three points: there is no central organizer, each panelist is grounded in a private corpus, and the panel is chosen by content rather than by persona.

A third group of papers constrains the design. Debate or Vote proves that unconstrained multi-agent debate is a martingale, meaning expected correctness does not rise across rounds. On Qwen2.5-7B it finds zero-round majority voting averages 0.7691, beating 2-round decentralized debate at 0.7377 and centralized debate at 0.6551. Voting or Consensus? finds that consensus beats voting on knowledge tasks (54.0% vs 51.7% on MMLU) but loses badly on multi-step reasoning (28.4% vs 54.8% on MuSR). It also shows that drafting independently before seeing peers reduces early conformity. Peacemaker or Troublemaker measures "disagreement collapse," where agents abandon correct positions to agree. It finds about 81% collapse in unmoderated decentralized debate on CommonsenseQA, which falls to 41% with a central judge, and it finds that the gains from debate run out after two or three rounds.

Together these results say that rounds of talk are risky and that the burden of proof is on anyone adding them. None of the three studies agents with different private evidence. That untested condition is the one our proposal targets. The older overlay literature provides the bounded neighbor lists and gossip-based link maintenance we build on, specifically top-$K$ "semantic acquaintances" in An Effective Schema Mapping Model for Decentralized Network and profile gossip in Topic 15 - Peer-to-Peer and Web Computing. The human Delphi studies motivate the anonymity and outlier-justification design (Dombrowski et al., Byrne et al.).

Core Claim

Let $p$ be the fraction of a question's panel whose private shard contains the decisive evidence. The predictions follow from the martingale argument in Debate or Vote, which applies whenever agents carry the same information. When $p$ is near 0, rounds should add nothing over a first-round vote, because nobody has the evidence. When $p$ is near 1, rounds should also add nothing, because everyone already has it. When the evidence is held by a minority, or when the answer requires composing facts from two different peers, round-two exchange can move correctness upward. In those cases Delphi-structured exchange should beat both a vote and open debate, because open debate lets the uninformed majority overwhelm the informed minority. The proposal is to build such a system, measure the accuracy gain from rounds as a function of $p$ and of whether cross-peer composition is required, and show where Delphi structure is and is not worth its messages.

Research Program

The system has three parts.

The first part recruits the panel. The asking node runs a TARW-style walk using the DRAG codebase. Instead of collecting snippets, it collects up to $n$ panelists, for example $n=7$, whose locally computed retrieval relevance for the question exceeds a threshold. Peers keep top-$K$ topical neighbor lists that are refreshed by gossip.

The second part is the anonymous first round. Each panelist retrieves from its own shard and answers without seeing anyone else. It returns three things: an answer, a confidence rating on a 1–9 scale, and a rationale of at most a fixed number of tokens. The rationale may state facts in the panelist's own words but may not quote documents. A sanitizer enforces an n-gram overlap cap against the panelist's shard. Rather than one peer acting as organizer, the panel computes an aggregate summary by gossip among its members, which is exact and cheap for panels this small. The summary contains the answer distribution, the median and interquartile range of confidence, and rationales with identities and ordering removed and deduplicated by embedding similarity, following the deduplication step of Bertolotti and Mari.

The third part runs the later rounds and the stopping rule. Panelists whose answer differs from the modal answer must either write a justification grounded in their shard or revise. All panelists may run one new local retrieval conditioned on the anonymized rationales. This step is what allows composition: peer B can search its own shard for something peer A asserted. Deliberation stops when agreement on the modal answer exceeds a preset threshold and the confidence interquartile range is at most one point, or after three rounds at the latest. The cap follows the collapse results. The final answer is the confidence-weighted mode. If agreement never reaches the threshold, the system abstains. This mirrors the filtering behavior that consensus showed on unanswerable questions in Voting or Consensus?.

The analysis has two parts. The first relates the round-two minus round-one accuracy gain to $p$ and to a composition flag. The second tracks disagreement collapse and the negative agreement rate, using the definitions from Peacemaker or Troublemaker, separately for informed and uninformed panelists. The mechanism we expect is that informed minorities hold their positions and uninformed majorities defer to them.

Baselines

The naive reference is a single agent at the asking node answering from its own shard alone. Next to it is the same agent with no retrieval at all. These two bound how much the network matters.

The strongest existing method under a no-raw-document constraint is zero-round voting over the same recruited panel, which is what Debate or Vote found hard to beat (0.7691 average on Qwen2.5-7B, on shared-context benchmarks). We run it in two forms: plain majority, and weighted by each panelist's self-reported confidence and retrieval score. Both use identical panels, models, and retrieval, so the only difference is whether rounds happen. Beating it means a statistically significant accuracy gain on the minority-evidence and composition slices, with no loss on the full-coverage slice.

The privacy-violating upper reference is DRAG-TARW itself, where snippets are shipped to the asker. Its reported result is 83.92% F1 on MMLU in Distributed Retrieval-Augmented Generation, and we rerun it on our partitions. Matching it within a few points while sending no document text would count as success.

We also run a centralized Delphi organizer modeled on Bertolotti and Mari, a central-judge variant that tests the 81%→41% collapse reduction reported in Peacemaker or Troublemaker, open decentralized debate with identified transcripts, and MAD-Conformist. Every method gets the same panel, the same per-question token and message budget, and the same prompt-tuning budget on a held-out split.

Ablations isolate each ingredient: a random panel in place of semantic recruitment, removing the outlier-justification step, removing the stopping rule, sending ratings only without rationales, and removing the second retrieval. The last ablation is the one that should kill the composition gain if our mechanism is right.

Why not just…?

Why not just take a confidence-weighted vote and skip the rounds? This is the real threat. When the decisive fact sits on a single peer, a well-calibrated weighted vote may already let that peer win, and Delphi rounds would then add only cost. We expect rounds to matter mainly on the composition slice, where no single peer can answer alone. If the weighted vote matches Delphi there as well, the honest conclusion is that semantic recruitment plus weighted voting is enough, and we will report that.

Why not just ship snippets, as DRAG does? When privacy allows it, snippet shipping is simpler and probably better, and we treat it as an upper reference rather than a competitor. The proposal targets settings where raw text cannot leave a node. We report rationale leakage as n-gram overlap with private shards so readers can judge whether short rationales are a meaningfully safer channel.

Why not just use a central judge? A central judge reduces collapse, but Debate or Vote reports centralized debate as its weakest configuration, and a central judge reintroduces a single point of failure and a single filter on what gets seen. Our centralized-organizer baseline uses the identical protocol with one aggregator. The comparison therefore measures what decentralization costs or buys in accuracy, independent of the protocol.

Why not just let agents debate freely? The martingale and collapse results already argue against it on shared-context tasks. Our decisive test is the informed-minority slice, where we predict open debate shows the highest negative agreement rate among informed panelists.

Proposed Experiments

We build partitioned versions of four benchmarks. For MMLU and the Medical Extended and News sets, we reuse the DRAG knowledge-base partitions so results connect to its reported numbers. For SQuAD 2.0, we split context passages across peers to test abstention on unanswerable questions. For StrategyQA, we split each question's supporting facts deliberately across two peers to create the composition slice. MMLU-Pro serves as a harder knowledge check. For each question we control $p$ by choosing how many panelists receive the evidence: none, one, about half, or all. Models are Qwen2.5-7B and Llama-3-8B, matching the debate papers.

Metrics are accuracy or F1, abstention precision on SQuAD 2.0, messages and tokens per question, disagreement-collapse and negative-agreement rates, and rationale leakage.

The decisive check is the curve of round-two gain against $p$. The hypothesis is supported if three things hold: the gain is near zero at $p=0$ and $p=1$, it is positive at intermediate $p$ and on the composition slice for Delphi, and it is flat or negative for open debate. The hypothesis is falsified if the confidence-weighted zero-round vote matches Delphi on the composition slice, or if the gain disappears when the second retrieval is ablated and nothing else explains it.