Scoring Expert Consensus Against Data: A Resolvable Delphi on Sex Differences in Cancer

Oncology is about to need many expert-consensus decisions about biological sex. Panels will be asked when a sex difference seen in mice or cell lines justifies separate male and female arms in a trial, and which sex-specific biomarkers deserve validation. The usual tool for such decisions is the Delphi method. In a Delphi study, experts rate statements anonymously over several rounds, see a summary of the group's ratings between rounds, and may revise their own. Its well-known weakness is that it measures whether opinion is stable, not whether it is correct. A cohesive panel can converge on a wrong assumption.

This proposal exploits an unusual property of sexual dimorphism in oncology: many of its claims can be checked cheaply against public data after experts have committed to a judgment. Examples include whether a drug hits female cell lines harder, whether a marker predicts survival only in one sex, and whether a mutation is male-enriched. We propose a Delphi study whose items are pre-registered, data-resolvable claims about sex differences in non-reproductive cancers. Each item is scored after the panel closes. This lets us measure directly whether iterative feedback moves experts toward the truth or merely toward each other. It also tests whether a cheap fix, feeding back partial empirical evidence between rounds, makes consensus more accurate. The output is both a calibrated consensus on sex-stratification questions and the first ground-truth audit of Delphi accuracy in molecular oncology.

Core Claim

Our central bet concerns the feedback a Delphi panel receives between rounds. Standard Delphi feedback reports the group median and interquartile range. On resolvable questions this mostly produces herding: ratings move toward the median whether or not the median is right. Adding a small, pre-specified piece of independent data evidence to the feedback should instead pull ratings toward the truth.

We write each expert's revision on item $i$ as

$$r_{i,2} - r_{i,1} = \alpha\,(m_{i,1} - r_{i,1}) + \beta\,(t_i - r_{i,1}) + \varepsilon_i,$$

where:

  • $r_{i,1}$ and $r_{i,2}$ are the expert's round-one and round-two probability ratings;
  • $m_{i,1}$ is the round-one panel median;
  • $t_i \in \{0,1\}$ is the later-resolved truth;
  • $\alpha$ measures pull toward the group;
  • $\beta$ measures pull toward the truth.

We predict that standard Delphi shows large $\alpha$ and near-zero $\beta$, and that evidence-grounded feedback raises $\beta$ without raising $\alpha$. If both arms show $\beta \approx 0$, iteration adds nothing to accuracy in this domain. That would be a useful negative result for every consensus guideline built this way.

Why This Intersection, and What Already Exists

Sex differences in tumor biology are now specific enough to generate checkable claims, and they often contradict the additive "adjust for sex" habit.

These papers motivate the proposal in two ways. They show that sex-specific effects are real and testable in PRISM, TCGA and PCAWG-style data. They also supply anchor items whose answers are already published.

On the Delphi side, the closest work constrains rather than duplicates us:

  • Metrics reloaded used a multi-round Delphi among 73 experts to reach 93% agreement on validation guidance for medical imaging. It reported agreement, never accuracy against an external truth, and contains no sex-related axis.
  • An LLM-based Delphi Study to Predict GenAI Evolution ran panels of GPT-4o-mini agents with distinct personas and reported 100% thematic consistency across 12 runs. Its authors note the models' positivity bias toward agreeable compromise, and consistency is again not correctness.
  • CALMA showed that structured deliberation surfaces disagreements that simple voting misses. We borrow its idea that the content of feedback matters, not only its form.

No paper in our search ran a Delphi on sexual dimorphism in oncology, and none scored Delphi convergence against data-resolved truth in a biomedical domain. Delphi studies of oncology consensus, such as Consensus on Molecular Subtypes of Ovarian Cancer, harmonize definitions rather than make checkable predictions.

Research Program

Item bank. We will build about 200 binary items from three public sources.

  • Drug response. Each item asks whether a named compound shows a sex-by-response interaction in the PRISM screen, within a named lineage, at a pre-registered false-discovery threshold.
  • Survival. Each item asks whether a named marker shows a sex-by-marker interaction on overall survival in a named TCGA cohort.
  • Genomics. Each item asks whether a named mutation or signature is enriched in one sex in TCGA or PCAWG-derived calls.

Items are written by a team separate from the analysts who resolve them, and the resolution code is frozen before elicitation. About 30 items are anchors with published answers (the Danusertib, TERT and glioblastoma findings). We use anchors only to measure expert knowledge, never in the accuracy headline. The remaining items have never been computed. This protects against both expert memory and LLM training-data contamination.

Handling sex labels. Cell lines frequently lose the Y chromosome, so we assign sex from XIST and Y-linked gene expression and drop discordant lines. Cell-line items are partly free of the gender confounders (smoking, diet, care access) that the PCAWG authors could not adjust for. Survival items are not, and we report the two strata separately.

Panel. We will recruit 60 to 80 experts across molecular oncology, bioinformatics and trial methodology, in a three-round Delphi. Experts give a probability for each item and a one-line rationale. Feedback arms are randomized at the item level within each expert. Every expert therefore contributes to every arm, which controls for individual skill.

The two main arms differ only in what experts see between rounds:

  • Standard Delphi: median, interquartile range and anonymized rationales.
  • Evidence-grounded Delphi: the same, plus one pre-computed independent data fragment. This is the same analysis run on a held-out half of the data: other PRISM lineages, a disjoint TCGA subsample, or a related cohort.

Resolution always uses the other half, so the fragment is informative but not the answer.

Baselines

The naive reference point is the round-one rating itself, aggregated by the panel median. This is a single anonymous survey with no iteration. If later rounds fail to beat it on Brier score (mean squared error of probabilities against outcomes), iteration buys nothing.

A second naive reference is the base rate: predicting the fraction of positive items within each source for every item. Any panel that fails to beat this has no item-specific knowledge.

The strongest competitor on accuracy is the evidence fragment alone, with no humans. We convert the held-out-half result into a probability using a logistic calibration fitted on the anchor items. This is the threat the proposal must survive. If the fragment alone matches the evidence-grounded panel, the experts add nothing beyond the data. We will run it on exactly the items and splits the panel sees. Beating it means a statistically lower Brier score in a paired bootstrap over items.

The standard-Delphi arm is the strongest existing consensus method. Run on the same items, experts and number of rounds, it isolates our key ingredient, the evidence in the feedback. A second ablation gives experts the evidence fragment but no peer feedback, separating the effect of evidence from that of iteration.

We also run a performance-weighted aggregate that weights each expert by their anchor-item accuracy. This tests whether simply trusting better-calibrated experts beats iteration.

Finally, we run an LLM-agent Delphi following An LLM-based Delphi Study to Predict GenAI Evolution: an organizing agent plus 5 to 15 persona agents over three rounds, in both feedback conditions. Its token budget is fixed so that agent and human panels see the same item text and the same fragments.

We found no reported Delphi accuracy-against-ground-truth result on Lacuna for any biomedical panel. The only reported Delphi figures are agreement and consistency rates: 93% expert agreement in Metrics reloaded and 100% thematic consistency in the LLM Delphi. We will report those statistics for our arms too, to show how weakly agreement tracks accuracy.

Why not just…?

Why not just compute every answer from the data and skip the experts? For the items in this study we can, and that is the point of choosing them. The decisions guideline panels actually face cannot be computed this way. Whether a mouse finding warrants a sex-stratified phase II arm has no dataset to resolve it. We use resolvable items to measure how trustworthy the Delphi process is before applying it to unresolvable ones. The "fragment alone" baseline tells us honestly whether experts add anything on top of data. If they do not, the right recommendation is that guideline panels defer to pre-registered reanalysis whenever data exist.

Why not just add sex as a covariate in a regression? Sex differences here are not additive shifts. In glioblastoma, opposite treatments work in each sex (Bayık et al.), and kinase networks diverge in lung adenocarcinoma (multi-omics LUAD study). That is why our items test sex-by-treatment and sex-by-marker interactions rather than main effects. More fundamentally, regression answers an analysis question. Our question is about the reliability of the judgment process that decides which analyses and trials to run.

Why not just run a single-round survey? This is a real threat. Delphi literature, such as the behavioral-trials Delphi in An international, Delphi consensus study to identify priorities for methodological research in behavioral trials in health research, asserts that controlled feedback improves judgment, but it measures convergence, not accuracy. Our round-one median baseline settles the question directly. If standard Delphi does not beat it, we will report that.

Why not just use an LLM panel and avoid recruiting humans? LLM panels are cheap and do not tire. Yet the only biomedical-adjacent evidence reports consistency rather than accuracy, and it flags a positivity bias that should inflate $\alpha$, the pull toward the group. Our LLM arms on never-computed items measure this directly. If LLM-plus-evidence matches the human panel, that is a strong practical result, and we would recommend it for piloting items.

Decisive Checks

The primary test is the paired difference in final-round Brier score between evidence-grounded and standard Delphi on never-computed items. The secondary test is the estimated $\beta$ and $\alpha$ from the revision equation in each arm.

The proposal's central claim fails if the evidence-grounded arm does not beat the fragment-alone baseline. It also fails if its gain over standard Delphi comes entirely from experts copying the fragment. We detect copying by comparing final ratings with the fragment-calibrated probability; true expert integration should beat both.

We will check that results hold separately on cell-line items, where gender confounding is minimal, and on TCGA survival items, where it is not. We will also check that agreement percentages diverge from accuracy, which would show that published consensus rates are an inadequate quality signal.

The deliverables are the following:

  • a resolved, released item bank with frozen resolution code, reusable by any future consensus or LLM-panel study;
  • a consensus statement on sex-stratification criteria, produced in the arm that proved most accurate.