Who Wins the Delphi Round? Role-Stratified Feedback for Forecasting Rural Land-Function Trade-offs

Rural planners in China and elsewhere increasingly "graft" farming onto tourism, cultural study tours and wellness services. A tea garden becomes a visitor attraction, and an embroidery tradition becomes a retail brand. Each graft trades off three things: what the land produces, how livable the village is, and how healthy its ecology stays. These trade-offs are usually settled by asking experts. The standard tool for this is the Delphi method, in which anonymous experts answer in repeated rounds and see a statistical summary of the group's answers between rounds. Delphi was built to stop loud or senior voices from dominating. Recent work on panels of language-model judges points to a quieter failure mode inside the method itself. When panelists hold asymmetric roles, such as a strict evaluator next to a lenient one, the shared numeric feedback pulls the group toward the strict pole and away from the truth. A rural grafting panel is asymmetric by construction: an ecological regulator, a tourism developer and a smallholder cooperative do not start from the same priors.

This note proposes to measure that pull on a task where the right answer is known, and to test a simple change to Delphi feedback that removes it without discarding the numbers. The task is forecasting. Panelists see a region's rural indicators up to 2010 and predict its 2020 production, living and ecological function scores. The test set is the 26-municipality Yangtze River Delta series, where the true trajectories are nonlinear and documented. The change is role-stratified feedback. Between rounds, each panelist sees the median of every stakeholder group separately, plus an equal-weight summary across groups, instead of one pooled distribution. The claim is that pooled feedback lets the most extreme role dominate the final consensus, and that stratified feedback restores roughly equal influence while keeping the calibration benefit of seeing numbers.

Core Claim

Three findings from prior work motivate the proposal.

  1. Role asymmetry biases consensus. Beyond Consensus: Downward Bias and Role Asymmetry in Multi-Agent LLM Judges studies essay scoring by debating language-model judges. On essay scoring it found that consensus debate with strict and lenient roles had a root mean squared error (RMSE) of 0.935 against human scores. A single judge scored 0.644. The consensus mean fell to 2.900 against a human mean of 3.486.
  2. Hiding the numbers is not the fix. The same paper found that hiding peer scores made coordination worse. Numeric feedback therefore matters, and its format is the variable to change.
  3. Early anchoring locks in fast. Emergence of Biased Consensus in Multi-Agent LLM Debates shows that low-temperature, high-conformity panels lock into their initial bias within one or two rounds, and that heterogeneity among agents softens this.

None of these papers studies a forecasting task with a delayed ground truth, a multi-dimensional target, or real stakeholder roles.

On the rural side, two kinds of study set the weights for land functions, and neither tests whether expert judgment adds anything over statistics.

- The nonlinear relationship between rural territorial functions and regional economic development derives weights by the Entropy Weight Method (EWM). EWM sets each indicator's weight by how much it varies across units in historical data. The paper reports weights of 0.40 for ecological, 0.35 for living and 0.25 for production functions. It also reports threshold effects: ecological function declines past roughly 12,000 yuan of GDP per capita, and overall functionality peaks near 13,000 yuan.

A Delphi forecasting backtest is the missing comparison. Its second contribution is a measurement of whose voice the consensus actually carries.

We define role influence directly. For each forecast target, the final consensus $c$ is regressed on the round-one medians $m_r^{(1)}$ of the $R$ stakeholder roles:

$$c = \sum_{r=1}^{R} w_r\, m_r^{(1)} + \varepsilon, \qquad \sum_r w_r = 1,$$

Here $w_r$ is the effective influence of role $r$. Equal voice means $w_r = 1/R$. The dominance gap is the largest $w_r$ minus $1/R$. The core prediction is that pooled feedback produces a positive dominance gap favoring the most conservative role, and that stratified feedback shrinks it without raising forecast error.

Research Program

Benchmark construction. The first component is the benchmark.

  • Indicators. We reconstruct the 12 production, living and ecological indicators for the 26 Yangtze River Delta municipal units, following the definitions in the Yangtze River Delta study. We source them from statistical yearbooks, or obtain them from the authors.
  • Targets. We compute the composite 2020 function scores as forecast targets. That gives 78 targets: 26 units times three functions.
  • Panel materials. Each unit gets a fixed briefing. It contains the 2000–2010 indicator series, its GDP trajectory, and a short narrative of local grafting activity. The narratives come from public plans in the style of the Qinhuangdao and Wuxi case studies.
  • Leakage controls. The targets are derived composites, not raw published figures, which limits memorization by language models. We also run a contamination probe: we ask models to recall the 2020 composites with no briefing.

Panels. The second component runs the same Delphi protocol with two kinds of panelist.

  • Language-model arm. Agents are given one of three stakeholder personas: ecological planner, tourism operator, or farm cooperative leader. This arm can be replicated hundreds of times. It lets us vary feedback format, panel size and sampling temperature, using the regimes identified by Okawa et al.
  • Human arm. This is a smaller panel of real stakeholders, roughly ten to twelve per role. They are recruited through agricultural, tourism and land-planning institutes and randomized between pooled and stratified feedback over three rounds. The human arm guards against a known concern, raised in AI Agents and the Future of Deliberation, that simulated deliberation may not reflect real deliberators. If the language-model and human arms disagree on the dominance gap, that disagreement is itself a reportable result.

Feedback formats. All other protocol details stay fixed. The formats are:

  • pooled median and interquartile range, which is standard Delphi;
  • role-stratified medians plus an equal-weight cross-role median, which is the proposal;
  • numeric scores masked, with rationales only, following the masking condition in Beyond Consensus;
  • no feedback at all, which is equivalent to averaging round-one answers.

Panelists forecast all three function scores jointly. This sidesteps the single-proposition limit acknowledged in Belief Engine and lets us see whether role dominance on the ecological target spills over into the others.

Downstream use. The third component is deliberately modest. If stratified feedback holds up, we apply it to a consensus "grafting fingerprint". This is a structured profile of a village's crops, ecological fragility, visitor capacity and heritage, which maps to recommended pathways. The template is the problem-fingerprinting procedure that Metrics reloaded built with a 73-expert Delphi reaching 93% consensus. Candidate items are filtered with Lawshe's content validity ratio, as in the clinical Delphi studies. This component has no ground truth. It is gated on the forecasting results and is reported only as an application.

Baselines

Statistical baseline. The naive reference is a purely statistical forecast that uses no experts. We fit the Kuznets-style inverted-U curves of the Yangtze River Delta study on 2000–2010 data only. We combine them with EWM weights computed from the same window and extrapolate to 2020. A per-unit linear trend is a second floor.

This is the strongest competitor on the forecasting question. The source paper shows these curves capture the documented thresholds, and the curves cost nothing to run. Both forecasts use exactly the briefing data panelists see. Expert consensus counts as beating this baseline only if its RMSE across the 78 targets is lower by a margin larger than the bootstrap confidence interval over units. No reported result found on Lacuna for out-of-sample forecasting on this series.

No-feedback crowd average. The second baseline averages panelists' independent round-one forecasts with no iteration. It isolates whether Delphi's feedback rounds help at all. It is a serious threat because, in Beyond Consensus, a single judge beat consensus debate on essay scoring: RMSE 0.644 versus 0.935. It is run on identical panelists and briefings.

Standard pooled-feedback Delphi. The third baseline is the strongest existing consensus method. It is the format used in the clinical and Metrics Reloaded Delphi studies. It is the direct comparison for the stratified format: same rounds, panelists and stopping rule. Stratified feedback beats it if two things both hold. The dominance gap must fall significantly. Forecast RMSE must be no worse, within a pre-registered non-inferiority margin of 5% of the pooled RMSE.

Masked-score format. This is the ablation isolating the key ingredient. If masking scores closes the dominance gap as well as stratification does, at no cost in error, then the benefit comes from removing anchoring rather than from restructuring feedback. The proposal's mechanism would then be wrong.

Sanity replication. In the language-model arm we first re-run pooled consensus on the NIKL essay set. We check that we recover the downward drift reported in Beyond Consensus before trusting our own role setup.

Why not just…?

Why not just use the Entropy Weight Method and fitted curves, with no panel at all? This is a real threat. The experiment settles it directly. If the statistical forecast matches the best Delphi format, the honest conclusion is that expert panels add little to forecasting these functions. Their value would then be limited to the forward-looking pathway fingerprint, which we would scope down accordingly.

The case for expected expert value is this. EWM and the fitted curves cannot see policy shifts or local operating realities absent from the historical variance. The Grain for Green evaluation discussed in The Mechanics of Qualitative and Quantitative Data Integration is an example: quantitative data showed ecological gains while missing farmers' economic distress. Whether that knowledge improves numeric forecasts is an empirical question, which is why this note does not assume it.

Why not just average independent answers and skip the rounds? Wisdom-of-crowds averaging is immune to role dominance by design, and single judges have already beaten debate on subjective scoring. If no-feedback averaging matches stratified Delphi on RMSE, then stratification only recovers what iteration broke. The practical advice would become "skip the rounds."

We include this arm because Beyond Consensus also found that numeric feedback aids calibration when it is present. Iteration may reduce error on targets where round-one variance is driven by misunderstanding rather than genuine disagreement. We test this per target by checking whether rounds help most where round-one spread is largest.

Why not just correct the consensus afterwards by subtracting each role's known bias? Post-hoc correction works when a ground truth is available for estimating the bias. Real planning decisions are forward-looking and have no such ground truth.

We test the alternative anyway. Role offsets are estimated on half the municipalities and applied to the other half, and separately estimated on the NIKL essays and applied to the forecasting task. If the offsets transfer, correction is a cheaper rival and we will say so. Our expectation is that the strict-pole pull depends on the target, for example on whether ecology is being forecast, so fixed offsets will not transfer.

Why not just use prompt optimization or memory masking from the multi-agent debate literature? CONSENSAGENT raised CLUTRR accuracy from 0.43 to 0.62 by rewriting prompts when debate stalls. Multi-Agent Debate with Memory Masking raised MATH accuracy from 71.2% to 80.2% by dropping low-confidence past reasoning. Both target reasoning tasks with a single correct answer and errors that can be detected along the way. Neither addresses legitimate role-based disagreement, and neither applies to human panels. We include memory masking as an extra language-model condition, to check that it does not already remove the dominance gap.

Proposed Experiments

Language-model study. We run three-round Delphi panels with nine agents, three per role. Panels are crossed with the four feedback formats and with two temperature regimes, low-uniform and heterogeneous, following Okawa et al. Each condition is replicated at least fifty times on the 78 targets. Outcomes are:

  • RMSE against the 2020 composites;
  • the dominance gap from the regression above;
  • rounds to reach a pre-set interquartile-range threshold.

Human study. This is a randomized two-arm comparison, pooled versus stratified feedback, on a 20-unit subset to limit panelist burden. It is pre-registered with the dominance gap as the primary outcome and RMSE non-inferiority as the secondary outcome.

Decisive checks. The main claim fails if pooled feedback shows no positive dominance gap in either arm. In that case the asymmetry finding from essay scoring does not carry over to stakeholder forecasting.

The main claim also fails if stratified feedback narrows the gap only by inflating error. For forecasting purposes, the panel question is settled against experts if the fitted-curve baseline beats every Delphi format.

The secondary diagnostic asks whether dominance concentrates on the ecological target near the 12,000–13,000 yuan threshold units. That is where the Yangtze River Delta study documents the sharpest trade-offs, and where a conservative ecological role would most plausibly pull the group.