When the Judge Can't Decide, Keep Both Branches: Order-Swap Disagreement as a Search Signal

Many recent systems make language models reason by search rather than by writing one answer straight through. The model proposes several possible next steps, something scores them, and the weak ones are discarded. When that "something" is another language model comparing two candidate steps side by side, the comparison carries a known flaw called positional bias: the judge tends to prefer whichever candidate appears first (or last), regardless of content. This flaw is worst when the two candidates are nearly equally good. Sibling branches in a reasoning tree are exactly that kind of near-tie, for example two plausible ways to apply the same theorem. Our proposal starts from this observation and asks two questions. How often does a search procedure discard the correct line of reasoning only because of the order in which two steps were shown to the judge? And can the judge's own inconsistency be used as useful information?

The core idea is simple. Show the judge each pair of sibling steps in both orders. If the two verdicts agree, prune as usual. If they disagree, read that as evidence the pair is close, and do not resolve it by averaging or by a coin flip. Keep both branches alive, and, when the budget allows, send the pair to a short, fixed-sides debate between two agents. Each agent defends one branch and must point to the specific domain rule it believes the other branch misapplies. We test this on theorem-heavy science and math problems, where models are known to lack the knowledge to apply formal principles correctly and where near-tied branches should be common.

Core Claim

Positional bias in a pairwise judge is not only noise to be removed. Within tree search, whether the verdict flips when the order is swapped is a cheap, label-free detector of near-tied siblings, and those are the nodes where pruning decisions are least reliable. Routing extra exploration to flipped pairs, rather than averaging them away, should recover correct branches that order-calibrated search throws out, at equal token cost. The claim is falsifiable at three points. Flipped pairs might not be enriched for small differences in true branch value. Keeping both branches might not improve final accuracy. A pointwise scorer with no ordering at all might do just as well.

Grounding in Prior Work

Three findings motivate the proposal. Judging the Judges studied 15 judges over 150,000 comparisons. It found that position bias is largely deterministic rather than random (top judges have repetition stability above 0.95) and that it dominates when the quality gap between candidates is narrow. It also found that the direction of the bias flips between general chat (recency) and software tasks (primacy), so it must be measured fresh in any new domain. Large Language Models are not Fair Evaluators showed that order alone can move a model's win rate from 51.3% to 23.8%. It proposed Balanced Position Calibration, which runs both orders and averages them, along with an entropy-based rule that sends uncertain cases to humans.

That second paper supplies the closest mechanism we build on. Its routing rule treats swap-disagreement as a reason for human review of a static answer. We treat the same disagreement as a reason to spend more search on a live reasoning tree. That changes what the signal is worth, because a wrongly pruned branch is unrecoverable while a wrongly retained branch only costs tokens.

Split and Merge (PORTIA) reduces order effects by interleaving aligned segments of the two candidates. It is an offline evaluation method. Here it serves as a strong debiasing competitor, not as a component.

On the search side, Policy Guided Tree Search learns when to expand, branch, backtrack, or stop, and reaches 41.0% on the MATH benchmark with Llama 3.1-8B against 34.4% for chain-of-thought. It scores states with log-likelihood, however, and it struggled on graduate-level science (GPQA). We read this as evidence that domain problems need a better node evaluator, and an LLM judge is the natural candidate. That is precisely where positional bias enters.

On the domain side, TheoremQA constrains the design. Open models score below 15%, and pasting theorem definitions into the prompt "did not provide a significant boost." Knowledge has to be checked at the point where a theorem is applied, not supplied up front.

The debate component is shaped by negative results. Three papers each document a way debate goes wrong. Peacemaker or Troublemaker ties sycophancy to agents abandoning correct positions (r = 0.902). When Debate Fails shows that debaters who share a flawed heuristic reinforce it. Voting or Consensus? finds accuracy drifts downward after 2–3 rounds.

We respond to all three in the same way. Debaters are assigned fixed sides they cannot leave, the debate is capped at two rounds, and its output is a ruling on one named step rather than a consensus. ChainLM debates every step of a linear chain. Tree-of-Debate builds a fixed topic hierarchy with no backtracking. Neither invokes debate only where a cheaper judge is unreliable, and neither attends to order effects.

Research Program

The work has three parts. The first is diagnostic and is worth publishing on its own. The search uses Llama 3.1-8B as the model that generates steps, so that results are comparable to Policy Guided Tree Search. On TheoremQA, MATH, and GPQA we grow reasoning trees and estimate a true value for every node. That value, $v(n)$, is the fraction of $K$ independent completions from node $n$ that reach the correct final answer, with $K = 16$.

For every sibling pair $(a, b)$ we then query an open-weight judge in both orders. We record the gap $|v(a) - v(b)|$ and whether the verdict flips. This tells us how flip rates depend on the value gap within reasoning trees, which no file paper has measured. It also tells us the direction of the bias in each domain, which matters because of the domain-dependent flips reported in Judging the Judges. Finally, it measures the "order-caused pruning rate": the fraction of problems where the only branch with $v > 0$ is discarded by a judge that sees one fixed order.

The second part is the search rule. A flipped pair is retained. A consistent pair is pruned. Under a fixed token budget, flipped pairs are expanded first, ahead of other open nodes. The third part is escalation. When budget remains and a flipped pair sits at depth $d \geq 2$, two agents run a two-round debate, each defending one sibling. A judge then rules only on whether a specific cited theorem or rule was instantiated correctly. To keep the debate itself from introducing an order effect, the speaking order is randomized and the ruling is run in both orders. If the ruling flips, both branches survive.

Baselines

The naive reference points are single-pass chain-of-thought and self-consistency voting over full chains at matched tokens. For chain-of-thought on MATH with Llama 3.1-8B, Policy Guided Tree Search reports 34.4%. On TheoremQA the file reports only that open models score below 15% and that GPT-4 with program-of-thought reaches 51%. Our 8B setting has no directly comparable number, so we rerun chain-of-thought ourselves.

The strongest existing search method is Policy Guided Tree Search, reported at 41.0% on MATH with Llama 3.1-8B at 5.28 times chain-of-thought tokens. We use that token ratio as the shared budget for every search variant. No reported result found on Lacuna for tree search of any kind on TheoremQA, and no GPQA accuracy number for Policy Guided Tree Search is given. We run it on both benchmarks with its released configuration.

The strongest debiasing competitor is the same tree search with a pairwise judge using Balanced Position Calibration from Large Language Models are not Fair Evaluators. It runs both orders, averages them, and breaks ties at random. This competitor sees exactly the same two judge calls per pair as our method, so any gain we show isolates what we do with the disagreement. A PORTIA-judged search (Split and Merge) is a second debiasing competitor.

Two ablations isolate the key ingredients. Flip-triggered retention without debate tests whether keeping both branches is enough. A random-retention control keeps the same number of extra pairs, chosen at random, to test whether flips carry information beyond simply widening the search. All methods share the generator, the judge model, the token budget, and a prompt-tuning budget of equal size on a held-out 10% split. Beating the competitors means a statistically significant accuracy gain over the Balanced Position Calibration search on TheoremQA and GPQA at matched tokens, plus a gain over random retention.

Why not just…?

Why not just average both orders, as Balanced Position Calibration does? This is the most direct threat, because it costs the same two calls. Averaging turns a flipped pair into an arbitrary tie-break. In search that means a near-tie is pruned about half the time, and a pruned branch can never be recovered. Whether this matters empirically depends on how often flipped pairs contain the only branch with positive value. The diagnostic measures exactly that, and the head-to-head comparison at equal calls and tokens settles it. If the order-caused pruning rate under averaging is negligible, the proposal fails.

Why not just score each step on its own, so order never arises? A pointwise scorer, such as a 1–10 rating or a process reward model, has no ordering and therefore no positional bias. This is a real alternative. Policy Guided Tree Search's GPQA failure shows that weak absolute scores are a problem in science domains, but it does not show that pairwise judging is better. We include a pointwise-judged search using the same judge model. If it matches our method, the honest conclusion is that pairwise judging should simply be avoided inside search.

Why not just widen the search beam? Retaining more nodes at random also protects correct branches. The random-retention control, at an equal number of retained pairs, tests whether flips direct the extra budget better than chance.

Why not just inject the theorem definitions? TheoremQA already reports that this gives no significant boost. We keep it as a cheap additive condition on all methods rather than treating it as a competitor.

Proposed Experiments and Decisive Checks

The first check is whether order matters in search at all. We run the single-order judge twice, once always presenting the left sibling first and once the right. A significant gap in final accuracy between those two runs establishes that order steers the outcome. The second check is enrichment. Flip rate should fall as the value gap $|v(a) - v(b)|$ grows. If flip rate is flat across gaps, flips are noise and the proposal is wrong. The third check is the head-to-head comparison against Balanced Position Calibration and PORTIA searches at 5.28 times chain-of-thought tokens on TheoremQA, GPQA, and MATH. We report per-subject results for TheoremQA (math, physics, EE&CS, finance), because near-ties should be most common where several theorems plausibly apply.

The fourth check is whether escalation helps. Debate escalation must beat retention-only on TheoremQA. We also log how often the fixed-sides debaters and the ruling judge agree with the value labels $v$, and we compare collapse rates against the levels reported in Peacemaker or Troublemaker. We expect retention, not debate, to carry most of the gain on MATH. We expect debate to matter only on theorem-application steps. If debate shows no added value there either, we drop it and report the simpler method.