Burden-of-Proof Debate for Irreversible Merges in Author Disambiguation

Scholarly databases must decide which papers bearing the name "Wei Wang" belong to the same person. Most systems do this by clustering: they score how similar pairs of papers are, then repeatedly merge the most similar groups. The costly mistakes are few and asymmetric. A single wrong merge of two different people fuses two separate co-author networks, and every later merge inherits that shortcut. Splitting one person into two profiles is comparatively cheap and easy to repair.

This note proposes spending a language-model debate only on the handful of merges that would be most damaging if wrong. A debate here means two or more model instances arguing opposite positions before a judge. The debate is built around that asymmetry. A "split" advocate searches the records for evidence that the two groups are different people. A "merge" advocate may win only by citing specific metadata fields, and a deterministic checker confirms those fields actually exist in the records. The default verdict is "do not merge." The claim is that this protocol cuts false merges at a fixed recall, where simply making the clusterer more conservative cannot. It also avoids the fabricated "shared mentor" stories that a single model reasoning step by step tends to invent.

Core Claim

Existing disambiguation pipelines handle uncertain merges with fixed rules or fixed thresholds. PNP's post-blocking step in Strong Baselines for Author Name Disambiguation with and Without Neural Networks forbids a merge when anchor papers lack overlapping relational evidence. The S2AND pipeline in S2AND: A Benchmark and Evaluation System for Author Name Disambiguation runs hierarchical agglomerative clustering with a distance cutoff. Neither can read the evidence when a researcher changes institution or field, so neither can tell that from a different person with the same name.

Multi-agent debate is known to help in exactly this kind of decision. In ERD: A Framework for Improving LLM Reasoning for Cognitive Distortion Classification, forcing an agent to defend the null hypothesis raised specificity from 6.79% to 30.74%. In MADAWSD: Multi-Agent Debate Framework for Adversarial Word Sense Disambiguation, debate beat single-model chain-of-thought (step-by-step written reasoning) when the context contained misleading cues.

Debate is also known to fail in ways that are fatal here. Examining Inter-Consistency of Large Language Models Collaboration: An In-depth Analysis via Debate shows that stubborn or flawed arguments can pull a stronger model off a correct stance. Peacemaker or Troublemaker: How Sycophancy Shapes Multi-Agent Debate documents agents conceding under social pressure. A symmetric debate that lets either side win by persuasion would therefore convert sycophancy into irreversible false merges.

The specific new claim is threefold. First, debate should be applied at the level of whole clusters, at the merge steps whose expected damage is highest. Second, the burden of proof should sit on the merge side. Third, every argument must be grounded in verifiable record fields. Under these conditions, debate improves the trade-off between avoiding false merges and recovering true ones beyond what threshold tuning, rules, or a single adjudicating model can reach.

Research Program

The base system is an unmodified disambiguation pipeline. On S2AND we use its released LightGBM pairwise model and average-linkage clustering. On WhoIsWho we use BOND from BOND: Bootstrapping From-Scratch Name Disambiguation with Multi-task Promoting, which learns paper embeddings with a graph attention network and clusters them with DBSCAN.

During agglomeration we intercept each proposed merge of two clusters and score its risk. The score is the base model's uncertainty, meaning how close the linkage distance is to the cutoff, multiplied by the number of paper pairs the merge would fuse. That product is a direct estimate of how many pairwise decisions a wrong merge would corrupt. Only the top-scoring merges per name block go to debate, under a fixed budget of language-model calls. Every other merge proceeds as the base model decides.

For each debated merge, both advocates receive a structured dossier for the two clusters. The dossier contains co-author lists, affiliation strings, venues, years, titles, and abstracts. Author names are stripped beyond the shared target name. This follows S2AND's "nameless" design and the WhoIsWho finding in Web-Scale Academic Name Disambiguation: the WhoIsWho Benchmark, Leaderboard, and Toolkit that adding names to text inputs hurt accuracy.

The split advocate looks for disqualifying evidence: simultaneous affiliations on different continents, disjoint co-author communities with no bridging paper, or incompatible career timelines. The merge advocate must produce bridging claims, and each claim must reference specific record IDs and fields. Two examples of valid claims are "paper 17 and paper 203 share co-author X" and "affiliation A in 2015 and affiliation B in 2016 suggest a move."

A deterministic checker verifies each referenced claim against the dossier and discards any claim it cannot verify. Fuzzy affiliation equivalence, such as "ICT" versus "Institute of Computing Technology, CAS," is allowed only as a labeled, lower-weight claim that the judge sees flagged.

The judge model rules after at most two rounds. That limit follows ERD's observation that gains saturate after two rounds. An early stop fires if the advocates agree, as with MADAWSD's consensus-seeker. The judge may approve the merge only if at least one verified bridging claim survives the split advocate's rebuttal. Otherwise the merge is blocked and the clusters stay separate.

The judge is the only place a language model makes a decision. No model-generated text enters the similarity scores. This respects the repeated finding, in BOND and in PNP's 30,100-parameter network, that large text representations can hurt clustering.

Baselines

The naive reference point is the base pipeline with no intervention. On WhoIsWho-v3 the strongest reported numbers are BOND at 87.72% F1 and BOND+ at 89.12% F1, both from BOND. The WhoIsWho toolkit's SND-all baseline reports 89.22% pairwise F1 (WhoIsWho). BOND+ is the strongest from-scratch competitor because it already tackles error accumulation between metric learning and clustering, which is the problem we target. For S2AND, the file gives only the relative claim of over 50% error reduction in B³ F1 against the Semantic Scholar production system, so we will re-run the released model ourselves. B³ F1 scores each paper by how much its predicted cluster overlaps its true cluster, then averages.

The strongest cheap competitor is threshold and rule conservatism. This means sweeping the clustering cutoff (and DBSCAN's radius for BOND) and adding PNP-style post-blocking. Both draw a full curve of false-merge rate against split rate. Beating it means the debate system's operating point lies strictly above that curve. In other words, it must have lower pairwise false-merge rate at equal recall, not just a different point on the same curve.

The strongest language-model competitor is a single adjudicator. One model gets the same dossier, the same evidence checker, the same default-to-split rule, and the same set of flagged merges. It is prompted to list reasons to split before deciding. We also run it with self-consistency voting, meaning several sampled answers combined by majority, scaled until its token budget matches the debate's.

Two ablations isolate the proposal's key ingredients. Symmetric debate removes the burden-of-proof asymmetry, so either side can win by persuasion. Unverified debate removes the checker. A random-selection ablation sends the same number of merges to debate chosen at random rather than by risk score. All variants use the same base model, the same judge model, the same tuning split, and a fixed call budget per name block.

Why not just…?

Why not just raise the clustering threshold? This is the most serious threat. Much of any precision gain could come from conservatism alone. The threshold sweep above settles it directly: if the debate's operating point falls on the conservatism curve, the proposal has failed. The argument that it can do better rests on S2AND's facet analysis, where errors concentrate on junior authors and recent papers. Those are cases where similarity is low for reasons unrelated to identity, such as a field change or a new affiliation. A threshold cannot separate those cases from true homonyms, but reading the records can.

Why not just use one model as judge with chain-of-thought? MADAWSD reports that chain-of-thought amplified distractor bias in disambiguation. ERD reports very low specificity for single-pass reasoning. Neither result was on bibliographic records, though. And a single judge given our checker and default-to-split rule might capture most of the benefit. The matched-budget single-adjudicator baseline tests exactly this. If it ties the debate, the honest conclusion is that verified burden of proof matters and debate does not, and we will report it that way.

Why not just add more hand-written rules? PNP's pre-merging rules alone reach 66.23 F1 on AMiner, and the full PNP reaches 70.19. Rules break, however, when metadata is missing. WhoIsWho reports a 36% performance drop without co-author names and a 10% drop without organizations. We test this by running all systems on blocks with co-author or affiliation fields artificially masked. The debate's advantage over rules should grow as metadata thins.

Proposed Experiments

The primary experiment is from-scratch disambiguation on WhoIsWho-v3 with BOND as base, and on the S2AND suite with its LightGBM base. We report pairwise F1 and B³ F1. We also report the pairwise false-merge rate and a cascade measure, defined as the number of wrong paper pairs created by each wrong merge decision. On S2AND we break results down by author prolificacy and publication year, to test whether gains concentrate on the junior-author cases.

A secondary experiment checks transfer to profile auditing on WhoIsWho-IND, where systems must find papers wrongly assigned to existing profiles. There the debated object is "does this paper belong to this profile." DIEq's reported gain of 0.5–1.2% weighted AUC (DIEq) is a relative improvement only. No absolute score is reported in the file, so we will re-run a LightGBM baseline.

Two checks are decisive. The first is whether the debate system strictly dominates the conservatism curve at matched recall on WhoIsWho-v3. The second is whether it beats the budget-matched single adjudicator. Two supporting checks clarify why. One compares the rate of fabricated claims rejected by the checker under debate versus the single adjudicator, a direct measure of the hallucinated-tie problem. The other measures how much each ablation changes the false-merge rate.

Cost is reported as model calls and tokens per name block, together with the fraction of merges sent to debate. This makes clear whether the benefit survives at budgets small enough to run across a whole database.