Expert and stakeholder panels decide many of the rules AI systems follow: when a clinical model should abstain, which outputs count as harmful, what a deployment policy permits. The usual tool for turning a panel's opinions into a rule is the Delphi method. Participants rate items anonymously, see the group's median and spread, and re-rate over several rounds until agreement stabilizes. Two things go wrong. First, "consensus" can be an artifact of who was recruited: a panel skewed toward one kind of person agrees on that kind of person's views. Second, the feedback rounds can push minority participants toward the median without persuading them. Statistical genetics faced the same structure decades ago. A variant looks associated with a disease only because both are enriched in one ancestral subgroup. The field's fix was to find continuous axes of variation in the data itself, using principal components (the main directions along which individuals differ), and to adjust for them.
This note proposes the same fix for consensus building. We estimate each participant's position on a few continuous "value axes" from their votes on concrete cases. We use those axes to re-estimate each item's consensus for a target population rather than for the recruited panel. Between rounds, we test whether opinion change is driven by axis position, which would indicate conformity, or is spread evenly, which would indicate genuine convergence. The result is a stratification-corrected Delphi protocol, together with a short screening set of cases that can recover a recruit's axis position cheaply. We validate it offline on the PRISM preference data and then in a prospective, randomized Delphi study that sets abstention rules for a clinical AI governance layer.
Core Claim
A Delphi consensus figure is a population estimate computed from an unrepresentative sample, and it can be confounded in the same way a genome-wide association is. The fix is not to collect demographic quotas. It is to (a) extract continuous latent axes from participants' case-grounded judgments, (b) report consensus standardized to a target distribution over those axes, and (c) flag items whose round-to-round convergence is explained by axis position. Our concrete prediction is that corrected consensus will reproduce the conclusions of a representative panel more faithfully than either naive or demographically reweighted consensus. A second prediction is that standard median-feedback Delphi produces measurable axis-aligned flattening, and that showing participants the axis-stratified distribution instead reduces it.
Why this intersection, and what already exists
Democratic deficit is the motivating problem here. Field studies show that top-down mandates fail when affected people are excluded. River-restoration satisfaction tracks participation and compensation equity more than water chemistry (Zhou et al.). Externally designed carbon programs are read locally as threats to land rights (Faisal). Two kinds of work show that participation can be operationalized but leave statistics aside. PolicyCraft has people vote on concrete cases and flags conflicts with their stated rules; it reported 74% majority support versus 23% for a discussion baseline among 66 students, using unweighted majority voting. NOGAPS and VISION-Cog use Delphi rounds and content-validity ratios to close clinical implementation gaps, with small panels and simple quota sampling. We reuse PolicyCraft's case-grounding as the input signal and the Delphi round structure as the protocol. What we add is the statistical layer both lack.
The ethical implementation gap supplies the application. ETHOS wraps a clinical multi-agent system with checks and an ethics critic. It raised decided accuracy from 0.674 to 0.712 by raising abstention from 26.7% to 39.3%, but its rubric was set top-down. Who decides those abstention thresholds, and whether that decision represents patients, is exactly where a corrected Delphi matters.
The closest technical work comes from pluralistic alignment. PRISM shows that model rankings change with rater demographics, that feedback from one group lowers out-group satisfaction, and that stated values predict contextual ratings weakly. It is observational and uses discrete demographic categories. Adaptive Pluralistic Alignment learns a low-rank basis of reward functions (LoRe) and aggregates personalized juror models with social-choice rules. Its jurors are simulated, and its votes are not adjusted for how the jury is composed. Pairwise Calibrated Rewards shows that ensembles of 2 to 4 reward models represent disagreement better than any single model, so a single aggregate has a floor on how well it represents a population. That last result constrains our design. We do not force one consensus number per item. We report a population-standardized estimate alongside an explicit test of whether the item is axis-dependent.
Population stratification correction supplies the method. Two pieces carry over directly. One is the principal-component covariate model. The other is the observation that a small, non-redundant marker set recovers the axes: 150 to 200 markers chosen by greedy QR column-subset selection recover European American ancestry axes at r > 0.9 (Paschou et al.). One warning also carries over. Over-parameterized adjustment can absorb real signal (polygenic overfitting). For that reason we standardize to a target population rather than regress the axes out, and we keep the number of axes small and chosen by held-out fit.
Research Program
The first component builds the axes. Each participant votes on a bank of concrete cases. In the clinical study these are vignettes such as "the model is 70% confident of cirrhosis, but the scan is degraded and the patient lives far from follow-up care: should it abstain?" The participant-by-case vote matrix is centered and factorized. Each participant $i$ gets a score vector $z_i \in \mathbb{R}^k$ on the top $k$ axes, with $k$ chosen by held-out vote reconstruction. Greedy QR column-subset selection then picks a short screening set of cases, around 20, from which $z_i$ can be predicted for new recruits. This plays the role of ancestry-informative markers and lets recruiters see a panel's axis coverage before round one.
The second component corrects consensus. For each Delphi item $j$ we fit $r_{ij} = \alpha_j + \gamma_j^\top z_i + \varepsilon_{ij}$, where $r_{ij}$ is participant $i$'s rating and $\gamma_j$ measures how strongly the item depends on the axes. We then report
$$\hat{c}_j = \frac{1}{|T|}\sum_{t \in T}\left(\hat{\alpha}_j + \hat{\gamma}_j^\top z_t\right),$$
where $T$ is a target population's axis scores. These come from a census-matched reference sample screened with the same cases. A large $\|\hat{\gamma}_j\|$ marks the item as contested along value lines. Such items are reported per region of axis space rather than collapsed into one number, in line with the floor result for single aggregates.
The third component diagnoses flattening. Between rounds, let $\Delta_{ij}$ be participant $i$'s rating change on item $j$, and let $d_{ij}$ be their round-one distance from the median. Ordinary regression to the median predicts that $\Delta_{ij}$ depends on $d_{ij}$. Conformist flattening predicts an extra dependence on $z_i$: participants on the minority end of an axis move more than equally distant majority-side participants. The coefficient on $z_i$ in a regression of $\Delta_{ij}$ on $d_{ij}$ and $z_i$ is our flattening index. We also check rationales: a shift without new reasoning counts as capitulation. As an intervention, we replace median-and-spread feedback with a display of the rating distribution split along the leading axis.
Baselines
The naive reference is standard Delphi: unweighted medians, interquartile range, and content-validity-ratio thresholds, computed on whoever was recruited, as in VISION-Cog. Every method sees the same panels and items. Beating it means closer agreement with the representative panel's item decisions.
The strongest simple competitor is demographic post-stratification, also called raking. It reweights participants so that self-reported age, gender, country, religion, and education match census margins, the categories PRISM records. This is what a careful survey methodologist would do, and it needs no new machinery. It gets identical data and target margins. Our method beats it only if axis standardization reduces error against the representative panel in skewed panels whose demographic margins look balanced but whose values are not, and does at least as well when margins are skewed.
The strongest existing latent-factor method is the LoRe low-rank reward basis from Adaptive Pluralistic Alignment, followed by its social-choice aggregation (Borda or Copeland). We run it on the same vote matrix with the same rank budget $k$ and the same hyperparameter search. A calibrated ensemble from Pairwise Calibrated Rewards, with 2 to 4 members fitted to the same data, serves as a second latent-structure competitor for representing disagreement. On PRISM, no reported result found on Lacuna for any of these methods recovering a representative-panel ranking from a skewed subsample. The only quantitative reference for the participatory setting is PolicyCraft's 74% versus 23% majority support. That measures support, not representativeness, so we treat it as context rather than a target.
Two ablations isolate the key ingredient. The first replaces the case-derived axes with axes built from participants' stated abstract values. This tests PRISM's stated-versus-contextual gap: if stated values work as well, case-grounding is unnecessary. The second replaces our axes with LoRe's factors inside our standardization step. This separates "which latent axes" from "standardize to a target," and we expect the second to be what matters.
Why not just…?
Why not just use demographic quotas or raking? Raking corrects only the categories it is given. The genomic evidence is that discrete labels miss continuous structure: European Americans form a gradient rather than clusters (Paschou et al.). PRISM's finding that rankings change across groups does not show that its categories capture the axes that matter. Raking is a real threat when values track demographics closely. Experiment 1 settles this directly by building skewed panels that are balanced on demographic margins but unbalanced on case-derived axes, and panels with the reverse property. If raking matches us on both kinds, the axis machinery is not worth it and we will say so.
Why not just use the LoRe low-rank basis already in Adaptive Pluralistic Alignment? LoRe already finds latent preference structure. Its pipeline, however, aggregates jurors' votes as given, so a skewed jury still gets its way. The low-rank basis is close to our axes, and we expect the two to perform similarly as axis extractors. Our contribution is the standardization to a target population and the flattening test, and the second ablation measures exactly that. If LoRe-plus-social-choice matches corrected consensus on representativeness, the standardization step adds nothing.
Why not just skip consensus and report the full distribution, or a pluralistic ensemble? For representing disagreement, a calibrated ensemble is more faithful (Pairwise Calibrated Rewards). A governance layer like ETHOS, though, needs an actual threshold, so the population-level estimate is still needed. We report per-region results only for items with high axis dependence. The test is whether practitioners can act on the rules this produces, measured as the fraction of items with a single deployable decision.
Why not just recruit a representative panel up front? Recruiting a representative panel is the ideal and is our ground truth. Over multiple rounds, however, dropout is non-random, which turns a representative panel into a skewed one. Correction is also needed for that reason. The screening set helps here too, by making axis coverage visible during recruitment.
Proposed Experiments
Experiment 1 is offline, on PRISM. We build a participant-by-model matrix of mean ratings. Each participant rated only a few of the 21 models, so the matrix is sparse, and we use matrix completion with held-out entries to choose $k$. The census-matched US and UK participants form the representative reference, and their model ranking is the ground truth. We draw thousands of skewed subpanels of 30 to 100 participants under three schemes: demographic skew, axis skew with balanced demographics, and simulated non-random dropout. Each method then estimates the reference ranking. Metrics are Kendall's tau with the reference ranking and agreement on the top model. We also compute PRISM-style out-group welfare: the average satisfaction of excluded participants under the chosen model. A decisive check is that corrected consensus must beat raking in the axis-skewed condition and must not lose in the demographic-skewed one.
Experiment 2 tests the screening set on PRISM and on the prospective case bank. We report the correlation between axis scores predicted from 10 to 40 QR-selected items and scores from the full item set, against randomly chosen item subsets of the same size. If the selected subsets do no better than random ones, the marker analogy fails.
Experiment 3 is prospective and randomized. We run a three-round Delphi to set abstention and escalation rules for an ETHOS-style hepatology assistant, using vignettes modeled on its fibrosis, cirrhosis, and fatty-liver decisions (ETHOS). About 300 participants take part: patients, carers, and clinicians, recruited deliberately unevenly, plus a separately recruited census-matched reference sample that completes only the case bank. Participants are randomized to standard median feedback or axis-stratified feedback. The primary outcomes are the flattening index and the rationale-coded capitulation rate across rounds. Secondary outcomes are the distance between corrected panel consensus and the reference sample's direct votes on held-out items, and whether participants accept the final rules. The decisive result is a positive flattening index under standard feedback that shrinks under stratified feedback. A null on both arms would mean that Delphi's conformity risk is less serious than its critics claim in this setting, and that finding would be worth publishing on its own.
References
- Analysis on Residents’ Satisfaction and Its Influencing Factors with Water Environment Management: Based on the Data from Xiaoqing River
- Local Perception Of The Carbon Emission Reduction Program In Oil Palm Plantations
- PolicyCraft: Supporting Collaborative and Participatory Policy Design through Case-Grounded Deliberation
- Bridging the gap between research evidence and real-world implementation: the nogaps project
- Feasibility, comprehensibility and acceptability of the VISION-Cog, a novel tool to assess cognitive impairment in visually impaired older adults: a cross-sectional pilot study in Singapore
- ETHOS: Towards a Modular Ethics Framework for Clinical Multi-Agent Systems
- The PRISM Alignment Dataset: What Participatory, Representative and Individualised Human Feedback Reveals About the Subjective and Multicultural Alignment of Large Language Models
- Adaptive Pluralistic Alignment: a pipeline for dynamic artificial democracy
- Pairwise Calibrated Rewards for Pluralistic Alignment
- Tracing Sub-Structure in the European American Population with PCA-Informative Markers
- Highly parameterized polygenic scores tend to overfit to population stratification via random effects