Multi-agent debate means several copies of a language model argue about an answer before settling on one. It works best when the agents take different roles, such as a critic facing an author, or a skeptic facing a believer. Those roles also bring a cost. When one judge is told to be strict and another lenient, the strict one pulls the group toward low scores. On Korean essay scoring this makes the debate less accurate than a single judge working alone. Our starting point comes from game theory. A game is symmetric when swapping the players' identities or roles leaves the payoffs unchanged. There is also a standard trick for making an asymmetric game symmetric: let chance decide who plays which role. Current debate protocols are asymmetric games that nobody has symmetrized.
We propose to do that symmetrization explicitly. The method starts from a set of transformations that should not change the correct answer: reflect the rating scale ("rate its quality" versus "rate its deficiencies"), negate the claim being defended, reorder the answer options, swap which model gets which persona, or swap who speaks first. We then measure how far a debate protocol's output moves under each transformation, after mapping the answer back into the original units. We call this movement the symmetry gap. It gives a label-free diagnostic that breaks debate bias down by source. It also suggests a repair. Run the role-asymmetric debate on a transformed copy of the task as well as the original, map each outcome back, and average. Any bias that flips sign under the transformation then cancels, while the argumentative friction the roles provide stays in place.
Core Claim
The strict-stance dominance reported in Beyond Consensus: Downward Bias and Role Asymmetry in Multi-Agent LLM Judges comes mainly from the valence of the roles, meaning which role argues for "worse." It does not come mainly from agent identity or from the content of the critique. If so, mirroring the scale should reverse the direction of the bias in native units, and averaging a debate with its mirror image should remove most of it. On that account the strict/lenient debate should come out as accurate as a neutral debate, or more so.
To state this precisely, let $x$ be a task instance, $f(x)$ the debate protocol's final answer, and $g$ a transformation with a known inverse that maps the transformed answer back to original units. The symmetry gap is $\Delta_g = \mathbb{E}_x\,|g^{-1} f(g x) - f(x)|$. The symmetrized protocol is
$$\bar f(x) = \frac{1}{|S|}\sum_{g \in S} g^{-1} f(g x),$$
where $S$ is a small set of transformations that includes the identity. Suppose the debate's error decomposes as $g^{-1}f(gx) = y^*(x) + e(x) + \sigma_g b(x)$. Here $y^*$ is the correct answer, $e$ is error that does not depend on $g$, and $b$ is a role-valence bias whose sign $\sigma_g = \pm 1$ flips under reflection. Averaging over a reflection pair then removes $b$ exactly. Whether $b$ actually behaves this way is the empirical question the project answers.
This differs from the closest work in three ways. Uncertainty-Aware Role-Switching Debate has debaters swap roles as one phase inside a single dialogue. That phase helps: dropping it lowers OpenBookQA accuracy from 74.3% to 65.0%. But the swap happens within one conversation, so what the agents said before the swap shapes what they say after it, and the paper offers no invariance measurement. We run independent mirrored debates and combine them outside the dialogue. Beyond Consensus fixes the bias by removing asymmetric roles. We keep the roles and remove their bias. From Self-Check to Consensus uses a formal signaling game to prevent collusion between agents, but it needs binary correct/incorrect states. The symmetry gap needs only a transformation with a known inverse, so it applies to 10-option questions and ordinal scores.
Research Program
The symmetric-game literature plays two roles in this project. It supplies the method: symmetrizing by randomizing roles is the textbook construction behind symmetric equilibrium analysis, of the kind used in Strong Mediated Equilibrium. It also supplies a warning. Systems tuned under symmetric self-play break down in asymmetric matchups, as in the StarCraft faction results of Huang et al.. This suggests that debate systems tuned with homogeneous agents may hide biases that only show up once roles differ. The decomposition of symmetric games into a common-interest part and a zero-sum part in Classification by decomposition gives us a lens. The part of a role's effect that flips sign under reflection is the "opposed" component we want to cancel. The shared component, meaning the extra scrutiny that any critic brings, is what we want to keep.
The work has three stages. The first builds the transformation sets for each task family. For subjective scoring (the NIKL Korean essays and SummEval), the transformations are scale reflection, persona-to-model swap, and speaker order. For multiple-choice reasoning (MMLU-Pro, OpenBookQA), they are option permutation, claim negation ("argue this option is correct" versus "argue every other option is wrong"), and persona swap.
The second stage measures $\Delta_g$ for every transformation across four protocols: a single judge, symmetric neutral debate, strict/lenient consensus debate, and actor–critic debate. The goal is to find out which symmetries each protocol breaks. We will also test whether the gap predicts per-item error without labels. As a process measure alongside the gap, we use the influence-asymmetry measure from A Diagnostic Study of Multi-Agent LLMs for Real-World Debates, which detects when a single agent dominates using an entropy-based score.
The third stage builds $\bar f$ with $|S|=2$, so it costs twice the calls of a single debate. It also builds an adaptive variant that runs the mirrored debate only when a cheap proxy predicts a large gap. We will use open-weight models of a single family at fixed size, so that persona effects are not confounded with differences in model scale, a confound Pitre et al. flag.
Baselines
The naive reference point is a single judge with chain-of-thought prompting. On NIKL essays, Beyond Consensus reports it at RMSE 0.644 against human scores. The strict/lenient consensus debate reaches 0.935, with a mean predicted score of 2.900 against a human mean of 3.486. We rerun both with our models and the same rubric. Symmetrized strict/lenient debate beats this baseline if it lowers RMSE below the single judge's rerun value while keeping the mean shift inside the noise level that we estimate by bootstrap.
The strongest competitor on the bias question is symmetric neutral debate, from the same paper. It is reported to "largely recover" single-judge alignment, but the file gives no reported number on Lacuna for Symmetric MAD's RMSE. We run it with the same number of rounds and the same total model calls as the symmetrized protocol, which means two independent neutral debates averaged together. Beating it at matched compute is the central test. If we cannot beat it, keeping roles has no justification on subjective tasks.
For multiple-choice truthfulness, the strongest existing method is role-switching debate at 74.3% on OpenBookQA (Liu et al.). We reimplement its five phases with our models and give it the same call budget. For multi-agent debate with trained roles, the relevant baseline is ACC-Debate (Estornell et al.), but the file has no reported MMLU number for it on Lacuna. We apply symmetrization on top of its actor–critic pair rather than competing against it, because the question there is whether trained asymmetric roles still carry a sign-flipping bias.
The key ablation separates the parts of the method. We compare symmetrizing only over persona-to-model swap, symmetrizing only over valence reflection, and symmetrizing over both. Our hypothesis predicts that valence reflection does nearly all the work. We also include a random-transformation control that averages two debates under transformations with no sign structure, such as paraphrases of the rubric. This separates a real cancellation effect from simple variance reduction by averaging.
Why not just…?
Why not just use symmetric neutral debate, as Beyond Consensus recommends? This is the most serious threat. The case for keeping roles rests on evidence that homogeneous agents fall into sycophancy and agree with errors, which ACC-Debate documents and which motivated its trained critic. Heterogeneous agent pools also show more engagement and balance, according to Pitre et al.. Those pools also showed lower group welfare, so the evidence cuts both ways. The matched-compute comparison above settles the question. If symmetric debate wins on essays but loses on MMLU-Pro, the honest conclusion is that roles pay off only where there is a correct answer to recover. We would report that result as it stands.
Why not just calibrate the strict/lenient debate by subtracting its average offset on a labeled dev set? A constant shift would fix most of the 0.586-point mean drop on NIKL, so this is a genuine threat on that metric. However, a constant shift needs human labels for each new rubric and domain. It also cannot correct bias that varies across items, for example bias that is larger for essays near the pass boundary. Mirroring needs no labels. We will run dev-set linear calibration as an extra baseline and compare per-item error and cross-dataset transfer, calibrating on NIKL and testing on SummEval. That comparison shows whether mirroring gives anything beyond what calibration gives.
Why not just sample the base debate twice and take a majority vote or average? Any two-run method might gain simply because averaging two runs reduces variance. The random-transformation control addresses this directly. If averaging two unmirrored debates improves accuracy as much as averaging a debate with its mirror, then the gain comes from averaging and not from symmetry, and the core claim is false. There is also evidence that outcome aggregation alone is a weak signal: Pitre et al. find that consensus barely predicts human preference, while healthy debate process recovers a wrong initial majority 74% of the time versus 40% for low-process debates.
Proposed Experiments
The first decisive check is purely diagnostic. On NIKL and SummEval we measure $\Delta_g$ for strict/lenient debate under each transformation. The hypothesis predicts that the reflection gap is large and that, measured in native units, the bias changes sign under reflection. If the bias instead keeps the same sign after reflection, then the model is harsh about content no matter how the scale is framed. In that case strictness is a content effect, cancellation fails, and we have learned that symmetrization cannot fix this failure.
The second check is the matched-compute comparison. Here we test mirror-symmetrized strict/lenient debate against the single judge, symmetric neutral debate, dev-calibrated debate, and the random-transformation control. The metrics are RMSE, mean shift, and Pearson correlation with human scores.
The third check extends the method to answers with ground truth. On MMLU-Pro we apply claim negation and option permutation to critic-versus-defender debates. For reference, MMLU-Pro reports GPT-4o at 72.6% as a single model. On OpenBookQA we compare against role-switching debate's reported 74.3%. We measure accuracy, and also the rate at which a wrong initial majority is recovered, using the same measure as Pitre et al.
The fourth check asks whether the gap measured on unlabeled items predicts which items the debate gets wrong. We test this with rank correlation on all four datasets, and we check whether a threshold on the gap can decide when running the mirrored debate is worth its extra cost.
Taken together, these checks answer whether the symmetry property that game theory relies on can be imposed on LLM debate after the fact, so that role diversity no longer comes with role bias.
References
- Beyond Consensus: Downward Bias and Role Asymmetry in Multi-Agent LLM Judges for Subjective Evaluation
- Uncertainty-Aware Role-Switching Debate: Improving Truthfulness in Large Language Models
- From Self-Check to Consensus: Bayesian Strategic Decoding in Large Language Models
- Strong Mediated Equilibrium
- A Robust and Opponent-Aware League Training Method for StarCraft II
- Classification by decomposition: a novel approach to classification of symmetric $$2\times 2$$ games
- A Diagnostic Study of Multi-Agent LLMs for Real-World Debates
- ACC-Collab: An Actor-Critic Approach to Multi-Agent LLM Collaboration
- MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark