When a panel of experts, or of language-model judges, revises its ratings after seeing what the others think, two different things happen. Some panelists change their minds because a peer's argument was good. Others move because being far from the group is uncomfortable. The Delphi method is the standard tool for structured expert consensus: anonymous ratings, repeated rounds, and feedback limited to summary statistics and anonymized rationales. It mixes both effects into one final number. Its known failure, a panel converging on a shared bias, comes from the second effect. Recent work on panels of language-model judges shows the same failure in sharp form. Lenient judges give way to strict ones, and the consensus drifts away from human ratings.
We propose to model each panelist's revision as a choice made by an altruistic agent, in the game-theoretic sense of someone whose utility includes how satisfied their peers are. Each submitted score balances the panelist's own post-evidence belief against peer dissatisfaction. With language-model panelists, the two forces can be pulled apart by experiment. We hold the anonymized rationales fixed and vary only the peer scores that are displayed, then do the reverse. This gives a per-panelist estimate of how much they concede and to whom, either the panel centre or the most dissatisfied peer. Inverting the fitted model recovers what each panelist believes after reading the arguments but before yielding socially. Aggregating those recovered beliefs should keep what Delphi gains from sharing information and remove what it loses to pressure.
Core Claim
Strict-stance dominance in multi-agent judging is a specific, identifiable form of altruism. Lenient panelists apply egalitarian (minimum-based) altruism: they concede toward the least-satisfied peer, who is usually the strict one. Strict panelists behave as selfish-first agents. The claim makes three predictions. The concession pattern can be measured with counterfactual probes. It is stable across items, so it behaves like a trait. Undoing it gives a panel estimate closer to human ratings than the Delphi consensus, the debate consensus, or the simple average of independent first-round scores. The mechanism is new. Prior debate work changes the protocol (who speaks when, when to stop), and altruistic-games work only studies static coalition formation. Nobody has used altruistic utility models as a measurement instrument on revision trajectories.
The Model
Altruistic hedonic games supply the vocabulary. Hedonic Games with Social Context adds an altruism weight on friends' payoffs to an agent's own valuation. Altruistic Hedonic Games distinguishes three degrees of altruism: selfish-first, equal treatment, and altruistic treatment. Stability in Minimization-Based Altruistic Hedonic Games replaces averaging over friends with the least-satisfied friend. The federated-learning study Does Egalitarian Fairness Lead to Instability? uses the same average-versus-worst-off split ("equal" versus "welfare" altruism) to bound how far a collective can bend toward its weakest member. All of these study partitions of agents into groups. We reuse the utility forms for a continuous rating revised over rounds.
Panelist $i$ holds a post-evidence belief $b_i$ about an item and sees anonymized peer scores $s_j$. The panelist submits the score $x_i$ that maximizes
$$U_i(x) = -(x-b_i)^2 - \alpha_i \,\mathrm{Agg}_{j\neq i}\,(x-s_j)^2,$$
where $\alpha_i \ge 0$ is the panelist's concession weight and $\mathrm{Agg}$ is either the average over peers (utilitarian) or the maximum over peers (egalitarian, since the least-satisfied peer is the farthest one). Under averaging, the submitted score is a linear shrinkage $x_i = (b_i + \alpha_i \bar s)/(1+\alpha_i)$ toward the peer mean $\bar s$. Under the maximum, the agent is pulled toward the midpoint of the farthest peer, and the response is piecewise and nonlinear in the extreme score. Selfish-first, equal, and altruistic treatment correspond to $\alpha_i$ near zero, near one, and large. The two operators make different predictions only when panels have more than two members, so we use five-member panels.
Research Program
We build a Delphi protocol for language-model judges that follows the classical design. In round one, panelists rate independently with rationales. In later rounds they see the median, the interquartile range, and anonymized rationales, as in the IBTN Delphi study and the ankle cartilage consensus proceedings. Panels mix persona types (strict, lenient, neutral) on top of a fixed model, which reproduces the asymmetry documented by Beyond Consensus. That paper motivates the work by showing the pathology. It does not provide a way to correct it.
Identification comes from counterfactual feedback, which is cheap for model panelists and impossible for humans. For a sample of items and rounds, we re-query each panelist under controlled feedback edits. The score probe keeps the real rationales but shifts the displayed peer scores, including only the most extreme one. The rationale probe keeps the scores but swaps in rationales from a different item, or adds a blank one. The submission's sensitivity to the score probe estimates $\alpha_i$ and selects the operator. The extreme-only shift separates the maximum from the average. The rationale probe measures persuasion. We then fit $(\alpha_i, \mathrm{Agg}_i)$ per panelist by least squares on the probe responses. We invert the fitted rule on the real, unperturbed trajectory to get $\hat b_i$, and we report the mean of the $\hat b_i$ as the panel estimate.
The analysis asks three questions. Are the fitted parameters stable across items? Does the egalitarian operator fit lenient panelists in strict-containing panels better than averaging? Does de-conceded aggregation move the panel mean back toward the human mean?
Baselines
The naive reference is a single judge, which in Beyond Consensus reaches RMSE 0.644 against human scores on the NIKL Korean essay set. The same paper's consensus debate degrades this to 0.935 and pulls the mean score from the human 3.486 down to 2.900. Beating 0.644 on NIKL, with the mean recovered to near 3.486, is the main success criterion.
The second baseline is the mean of independent round-one scores. It contains no social pressure and also no shared evidence, so it is the most direct threat; see below.
The third is classical Delphi: the median after three rounds of median-plus-IQR feedback, with no modeling.
The fourth is the strongest existing deliberation method on judge benchmarks, the collaborative debate with Beta-Binomial/Kolmogorov–Smirnov adaptive stopping from Multi-Agent Debate for LLM Judges with Adaptive Stability Detection. It reports LLMBar accuracy of 81.83% versus 77.75% for static majority voting with Gemini-2.0-Flash. For SummEval, no reported result was found on Lacuna for exact RMSE or correlation values. We will run all baselines ourselves there.
We also include the voting protocols and All-Agents Drafting from Voting or Consensus? as protocol-level fixes for sycophancy.
Fairness rules apply to all comparisons. Every method uses the same base model, personas, and items. Our probes cost extra model calls, so every baseline gets the same total call budget, for example more round-one samples or more debate rounds. Each method gets the same validation split for tuning any thresholds.
Two ablations isolate the key ingredient. The first inverts the trajectory with one shared $\alpha$ fitted across all panelists, which is plain DeGroot-style shrinkage removal. The second forces the averaging operator for everyone. If per-panelist, operator-aware inversion does not beat these ablations, the altruism model adds nothing beyond a correction constant.
Why not just…?
Why not just average the independent first-round scores? This is the real threat. Sharing information may simply not help subjective scoring, in which case everything after round one is noise. Our claim needs $\hat b_i$ to beat the round-one mean at matched budget. The rationale probe shows directly whether real rationales move beliefs more than mismatched ones. If they do not, we will report that Delphi feedback carries no usable evidence for these tasks. That would itself be a useful negative result for the deliberation literature.
Why not just prompt the judges to be neutral or cooperative? Personas are themselves prompts, and Beyond Consensus shows that prompted stances produce the asymmetry rather than control it. Voting or Consensus? reports sycophancy and drift under conversational interaction. We include a "be balanced" prompt arm. If it removes the measured difference in $\alpha$ between strict and lenient judges, the prompt is the cheaper fix and we will say so.
Why not just hide the scores so nobody can concede to them? Beyond Consensus found that masking numerical scores made coordination worse, so scores carry calibration information. Our approach keeps the scores visible and removes only their estimated social component.
Why not just stop earlier with adaptive stability detection? Stopping rules decide when the distribution has stopped moving, not whether it moved for good reasons. Hu et al. note a bimodal pattern in which some debates converge on collective failure. A stable consensus pulled down by strict panelists would pass a Kolmogorov–Smirnov test unchanged.
Decisive Experiments
We run five-member panels on NIKL essays (600 essays, three rubric dimensions) and SummEval (700 summaries, four dimensions), scored by RMSE, Spearman correlation, and mean bias against human ratings. We use LLMBar, read as preference probabilities, to check that the correction does not hurt objective judging.
The first decisive check is trait stability. The rank correlation of each panelist's fitted $\alpha_i$ between disjoint halves of the items must be clearly positive. If it is not, the utility model describes no stable behavior and the project stops there.
The second is operator identification. The extreme-only score probe should move lenient panelists more than strict ones, and the maximum operator should win held-out fit for lenient panelists. The averaging operator should win for neutral panelists.
The third is accuracy. De-conceded aggregation should beat the single-judge RMSE of 0.644 on NIKL, the round-one mean, classical Delphi, and adaptive-stopping debate at matched call budget. It should also beat both ablations.
As a secondary, exploratory step, we will fit the same model to human Delphi panels where per-round individual ratings are available. Without probes, identification there relies only on natural variation in feedback, so we treat it as descriptive.
References
- Hedonic Games with Social Context
- Altruistic Hedonic Games
- Stability in Minimization-Based Altruistic Hedonic Games
- Does Egalitarian Fairness Lead to Instability? The Fairness Bounds in Stable Federated Learning Under Altruistic Behaviors
- An international, Delphi consensus study to identify priorities for methodological research in behavioral trials in health research
- Conservative Management and Biological Treatment Strategies: Proceedings of the International Consensus Meeting on Cartilage Repair of the Ankle
- Beyond Consensus: Downward Bias and Role Asymmetry in Multi-Agent LLM Judges for Subjective Evaluation
- Multi-Agent Debate for LLM Judges with Adaptive Stability Detection
- Voting or Consensus? Decision-Making in Multi-Agent Debate