Practitioners increasingly grade math and science answers with a panel of language-model judges. The panel is often one model prompted with several personas, such as "a logician," "a physics grader," or "a skeptical reviewer." They then trust the verdicts the panel agrees on. That trust borrows an assumption from human annotation studies: when annotators agree, it is because each of them independently got it right. LLM personas break this assumption. Earlier work found that different personas of the same model agree with each other extremely highly, which suggests one reasoning template sitting underneath all the costumes. If so, panel agreement on hard items may be agreement on the same mistake. This proposal uses symbolic equivalence checking as a measuring instrument. Symbolic equivalence checking means computer algebra or a logic solver deciding whether two expressions mean the same thing, regardless of how they are written. It supplies certified true answers for answer-equivalence judgments. With those answers we can measure how much judge agreement overstates judge correctness, compare panels built from personas, from repeated sampling, and from different model families, and calibrate when a panel's consensus can stand in for the solver on items the solver cannot handle.
The two-sentence version is this. Agreement among LLM judges is only evidence of correctness if their errors are roughly independent, and in formal domains we can test that directly because a solver tells us the right answer on many items. We will quantify the "agreement inflation" of persona panels and use it to build a deferral rule. The rule decides when to trust the panel and when to send an item to a solver or a human, and it should beat raw vote margins at matched coverage.
Core Claim
Inter-annotator agreement measures such as Cohen's or Fleiss' kappa and Krippendorff's alpha correct for agreement that would happen by chance. They do not correct for annotators who share the same blind spot, which is the dominant failure mode for prompted variants of one model. We claim three things. First, on answer-equivalence tasks, persona panels show substantially more agreement on wrong verdicts than their individual error rates would predict under independence. Second, this inflation is larger for persona panels than for panels mixing model families. Third, an estimate of inflation, made on solver-certified items, gives a better trust signal on uncertified items than either the panel's raw vote margin or its chance-corrected agreement.
The key quantity is easy to state. Suppose panel members $j = 1..k$ have individual error rates $e_j$ on binary equivalent/not-equivalent verdicts. If their errors were independent, the probability that a unanimous verdict is wrong would be
$$\hat{q}_{\text{ind}} = \frac{\prod_j e_j}{\prod_j e_j + \prod_j (1-e_j)}.$$
We define agreement inflation as $I = q_{\text{obs}} / \hat{q}_{\text{ind}}$, where $q_{\text{obs}}$ is the observed wrong-when-unanimous rate on certified items. A value $I \gg 1$ means agreement is mostly shared error. The formula is Bayes' rule over the panel's joint verdict with a balanced prior. We will use the empirical label balance in practice.
Where This Sits in Prior Work
Three lines of work motivate the question. Persona Prompting as a Lens on LLM Social Reasoning applied 21 single-attribute and 12 composite personas to subjective classification. It measured inter-persona Krippendorff's alpha of at least 0.80 for two of three models and found that personas degrade rationale quality even when labels improve. It never had ground truth for whether the agreement was correct, and it did not study judging. We supply that ground truth. Beyond a Single Judge: Simulating Social Persona Panels for Generative UI Evaluation showed that a five-persona deliberating panel tracks human ratings of UI quality far better than a single judge (Pearson 0.922 vs. 0.716). Its target is subjective, and it reports correlation with human means rather than whether the panel's internal consensus is reliable. We ask the complementary question in a domain where correctness is objective, and ESPP's positive result is the claim our experiments may contradict for formal judging. JURY-RL found that unverified majority votes over rollouts make models "confidently wrong." That is the same correlated-error phenomenon seen through reinforcement learning. Its fix was a Lean verifier, which it reports at 84.5% precision against 75.9% for an LLM judge.
Two papers constrain the design. Risk-Controlled Lean-as-Judge for Natural-Language Mathematical Reasoning found that only about 43% of kernel-accepted Lean proofs were faithful to the original problem, and that its COVCAL selective predictor certifies only about 48% of MATH-500 problems at 0.98 accuracy. Two consequences follow. Theorem-prover acceptance cannot by itself be our ground truth, and there is a large uncertified remainder where a well-calibrated panel would be useful. We therefore build ground truth from answer-level equivalence checked by computer algebra and SMT solvers on certified rewrites, not from end-to-end Lean proofs. SCI-Verifier shows a trained 8B verifier reaching 92.72% on SCI-VerifyBench against 88.92% for GPT-5. A learned verifier is therefore a strong competitor to any prompted panel and must be a baseline.
Three papers supply method. Autoformalize Mathematical Statements by Symbolic Equivalence and Semantic Consistency groups candidates by solver-checked equivalence with variable matching. PRISM-Physics checks solution-set equivalence by normalizing units and substituting random values. Rethinking and Improving Autoformalization contributes the BEq bidirectional-proof metric and a 200-pair expert-labeled Lean equivalence set. On that set LLM judges score 48–82% and BEq scores 90.5% accuracy with 100% precision. We reuse these checkers to certify labels, and we do not propose a new checker.
Research Program
The first component is a certified judging testbed. We start from the mathematics, physics, and chemistry portions of SCI-VerifyBench, and from MATH-500 reference answers. From these we generate candidate answers in two ways. Equivalent candidates come from computer-algebra rewrites: factoring and expanding, unit conversions, rationalizing, and trigonometric identities. Non-equivalent candidates differ by a single sign, exponent, operator, or unit-prefix change. These are exactly the cases where string overlap is high and meaning flips. Each pair carries a SymPy or Z3 certificate of its label. Because rewrites are applied to the same base items that judges will later see uncertified, the certified set is not just the "easy" subset. This is our main defense against the shift between solver-checkable and solver-uncheckable items. We stratify by rewrite depth so we can see whether inflation grows as surface similarity and meaning come apart.
The second component compares four kinds of judge panel at an equal number of judge calls, $k = 5$. The first is one model sampled at temperature. The second is one model under five fixed personas: formal logician, physics grader, computer-algebra-minded engineer, adversarial critic, and lenient teacher. The third is five model families under a neutral prompt. The fourth mixes personas across families. For each panel we record per-judge accuracy, Krippendorff's alpha, pairwise error correlation, and the inflation $I$ by rewrite stratum. These are the core scientific measurements. They answer whether persona diversity buys independent errors at all.
The third component is a deferral rule. On a calibration split of certified items, we fit the mapping from a panel's verdict pattern to its empirical error rate within each stratum. The pattern is unanimous, 4–1, 3–2, together with which personas dissent. We then set thresholds that guarantee a target error rate with a finite-sample bound, in the style of COVCAL's calibration. Items below threshold go to the solver if it applies, or otherwise to abstention. The rule is evaluated on held-out certified items and then transferred to the uncertified MATH-500 remainder that COVCAL abstains on.
Baselines
The naive reference point is a single strong judge prompted once with a rubric, using its stated confidence as the trust signal. On FormL4-Basic, GPT-4 as a single judge reached only 42.68% precision according to FormalAlign. Any panel that fails to clearly beat this is uninteresting.
The strongest prompted competitor is self-consistency. It uses the same model, five temperature samples, and the vote margin as confidence. It is cheap and directly targets the same signal, so it is the comparison most likely to embarrass the proposal. It gets the identical call budget, the same prompt template minus the persona line, and the same calibration split for threshold fitting.
The strongest existing method on raw verification accuracy is SCI-Verifier-8B, which reports 92.72% overall accuracy on SCI-VerifyBench against 88.92% for GPT-5 and 80.16% for xVerify-8B, all from SCI-Verifier. We run it on our perturbation set as released. It is not retuned, since retuning would require its training data, which we do not alter. We compare at matched coverage on selective accuracy, using its own output probability as the confidence signal. On the uncertified math remainder, the reference is COVCAL, which reports about 48% acceptance at 0.98 certified accuracy on filtered MATH-500 with Goedel-Prover-V2-8B, from Risk-Controlled Lean-as-Judge. On the Lean equivalence task, the reference is BEq's 90.5% accuracy and 100% precision on the 200-pair human set, from Rethinking and Improving Autoformalization.
The ablation that isolates our key ingredient replaces the inflation-aware, pattern-specific calibration with two alternatives. One is the panel's raw percent agreement. The other is its chance-corrected agreement. Both use the same panels and thresholds. If inflation-aware calibration does not beat these, the contribution reduces to measurement alone.
Beating a baseline means one of two things. The first is lower held-out error at equal coverage, with the risk curve's confidence interval excluding the baseline's. The second is that on MATH-500 the deferral rule accepts more than COVCAL's roughly 48% of problems while holding held-out accuracy at or above 0.98 on a freshly hand-graded audit sample.
Why not just…?
Why not just use self-consistency vote margin and skip personas? This is the real threat. If persona prompts only add a costume over one reasoning template, as the high alpha values in Persona Prompting as a Lens on LLM Social Reasoning suggest, then persona panels should show the same inflation as temperature sampling. The cheaper method would then win. Our design makes this a measured outcome rather than an assumption. If persona and temperature panels have indistinguishable inflation and risk curves, we will report that persona prompting buys nothing for formal judging. We will then recommend the self-consistency or cross-model panel, and the inflation-aware calibration stays as the contribution. The evidence for some diversity effect comes from ESPP. There, averaging five identical calls closed only about a third of the gap to human ratings, while distinct grounded personas closed most of it. That was in a subjective domain, however, and may not carry over.
Why not just use a panel of different models instead of personas? It may well be better. Different model families plausibly fail in less correlated ways than prompts on one model. That is why cross-model panels are one of our four conditions and not an afterthought. The measurement of $I$ per panel type directly answers this, and the cost of hosting several families is reported alongside the accuracy.
Why not just run the solver on everything, or use a trained verifier? Solvers do not cover everything. Half of MATH-500 goes uncertified under COVCAL, and kernel acceptance is often unfaithful. The panel is valuable exactly where the solver is silent. A trained verifier like SCI-Verifier-8B is strong and is included as a baseline. Its weakness is that, being a single model, it has no internal disagreement signal. Its self-reported confidence may suffer the same overconfidence on single-sign flips. Matched-coverage risk curves on the perturbation strata will show whether it does.
Why not just use chance-corrected agreement as the trust signal? Chance correction removes agreement expected from label frequencies. It says nothing about shared blind spots, so a panel can have high alpha and high inflation simultaneously. The ablation above tests whether this distinction matters in practice.
Decisive Checks
Three results decide the project. The first asks whether persona panels have inflation significantly above 1 on deep-rewrite and single-flip strata, and whether that inflation exceeds cross-model panels'. If neither holds, the correlated-error story is wrong. The second asks whether the inflation-aware deferral rule beats self-consistency vote margin and chance-corrected agreement at matched coverage on held-out certified items. If not, the practical contribution collapses to a measurement study. The third asks whether calibration fitted on certified rewrites transfers. We test this on the BEq 200-pair expert set, which was not selected by solver coverage, and on a hand-graded sample of the MATH-500 items COVCAL abstains on. Here we will also re-annotate 100 items with two independent experts and report Cohen's kappa between them, so that the human reference itself has a known reliability. If transfer fails, we learn that solver-checkable items are systematically unrepresentative, which is itself a useful warning for every neuro-symbolic evaluation pipeline that calibrates on its covered subset.
References
- Persona Prompting as a Lens on LLM Social Reasoning
- JURY-RL: Votes Propose, Proofs Dispose for Label-Free RLVR
- Risk-Controlled Lean-as-Judge for Natural-Language Mathematical Reasoning
- SCI-Verifier: Scientific Verifier with Thinking
- Autoformalize Mathematical Statements by Symbolic Equivalence and Semantic Consistency
- PRISM-Physics: Causal DAG-Based Process Evaluation for Physics Reasoning
- Rethinking and Improving Autoformalization: Towards a Faithful Metric and a Dependency Retrieval-based Approach
- FormalAlign: Automated Alignment Evaluation for Autoformalization