Outcome-Anchored Delphi for Judging Oocytes from Older Women

Embryologists decide which eggs and embryos to culture, transfer or discard largely by eye, and the criteria they use were built mostly on younger patients. Age is the strongest single predictor of failure in assisted reproduction. Mechanistic work shows that aged eggs fail in ways a microscope barely sees: sugar shunted into sorbitol, mitochondria under oxidative stress, softened cell cortices that crowd the cytoplasm and block chromosome capture. So the question of what a "good" egg from a 41-year-old looks like is open, and nobody has tested it. The usual tool for settling such questions is the Delphi method, in which experts rate statements anonymously over several rounds and see the group's aggregated answers between rounds. Delphi measures agreement, though, not truth, and a panel can converge confidently on a wrong rule.

We propose a Delphi in which the experts see, between rounds, both how their peers voted and how well each proposed criterion actually predicted embryo outcomes in a held-out set of cycles, broken down by maternal age. We split one expert pool at random into two parallel panels, one with this outcome feedback and one classic, and run a third panel made only of language models. Then we score the criteria each panel produces on a second, untouched cohort. The point is to learn whether evidence feedback inside the consensus loop yields age-conditional quality criteria that predict outcomes in older patients better than classic consensus, individual graders, a purely statistical model fit to the same data, or a language-model panel.

Core Claim

Expert criteria for oocyte and embryo quality are least reliable where data are scarcest, in women aged 38 and over. We expect that adding empirical, age-stratified calibration feedback to Delphi rounds will change which criteria reach consensus. We also expect the resulting rules to predict blastocyst formation and euploidy better in that stratum than classic Delphi rules do. The new part is not running a Delphi on embryology, which on its own would be routine. The new parts are two. First, outcome feedback is treated as a manipulated variable, tested in a randomized two-arm design. Second, the products of consensus are judged against held-out biological outcomes rather than against the panel's own agreement.

Why this intersection, and what already exists

Machine-learning work on gametes and embryos inherits its labels from subjective grading. One study trained a random forest on hand-assigned A–D grades of cumulus-oocyte complexes from 80 pig oocytes. It reached 88.35% accuracy, with errors concentrated between the intermediate grades B and C (Grading of mammalian cumulus oocyte complexes using machine learning). That is the signature of a label that experts themselves cannot draw sharply. A multi-task network trained on 249 single-clinic human blastocyst images with embryologist-assigned Gardner grades scored a trophectoderm F-score of 0.64 and an expansion F-score of 0.76. Shared training did not help inner cell mass grading (MEmEBG). A DINOv2-plus-attention-LSTM model on 704 human time-lapse videos predicts blastocyst formation with 96.4% accuracy and an AUC of 0.99. It treats the outcome as binary and ignores maternal age (Predicting Blastocyst Formation in IVF). These papers motivate the proposal: they show the label problem, and they show that age is absent from the modelling.

The consensus side supplies the method. Metrics Reloaded used a 73-expert multi-stage Delphi, reaching 93% agreement, to standardize how image-analysis metrics are chosen. It included a sperm-motility use case but stopped at metric choice and did not address what counts as ground truth (Metrics reloaded). FUTURE-AI ran eight Delphi rounds with 117 experts on general deployment principles for medical AI, with no domain-specific criteria (FUTURE-AI). The Human–AI Hybrid Delphi is our closest methodological neighbour (HAH-Delphi). It pairs Gemini 2.5 Pro, working over a constrained literature corpus, with small senior panels and offers a four-level taxonomy that includes "conditional consensus." It reports that the model reproduced 95% of historical consensus items, and that panels of six reached thematic saturation. Its feedback is literature-grounded, however, and never outcome-grounded. Its success is measured as agreement with past panels, and its authors note an "experience gap" in the model's reasoning. We borrow its conditional-consensus category, because age-dependent rules are exactly "it depends" rules. We differ from it in what enters the loop (calibration against data) and in what counts as success (prediction of held-out outcomes).

The biology constrains which statements are worth rating. In aged mice, the drug Sorbinil cut sorbitol from 14.08 to 0.48 ng per oocyte and raised the rate at which eggs extrude their first polar body (the visible sign that the first meiotic division finished) from 66.1% to 82.4%. It did not restore blastocyst development to young levels (Sorbitol accumulation decreases oocyte quality in aged mice). Polar-body extrusion is a standard visible maturity marker, so this result predicts that experts will overweight it in older patients. That is a concrete case where outcome feedback should move consensus. Metformin rescue in aged mice also improved spindle and chromosome alignment, which dropped from 64.6% to 38.19% defective (Metformin promotes in vitro maturation of oocytes from aged mice). In parallel, about 36% of mouse oocytes are "extra-soft," and softening alone raises misalignment from 15% to 48% with normal-looking spindles (Artificially decreasing cortical tension generates aneuploidy). Together these mean that some aneuploidy risk may show up as cytoplasmic or shape cues rather than as spindle defects. We include such cues as candidate items, knowing they may not be visible in routine imaging. Paternal age also interacts with maternal age in implantation failure across 1,910 couples (adjusted OR 2.13 for paternal age ≥45) (Effects of paternal age and interaction with smoking). We therefore allow items that condition on both parents.

Research Program

We first build an item bank of roughly 60 candidate statements. Each statement is a scoreable criterion that a grader or an image pipeline can apply to an oocyte or embryo. Each has an explicit age condition where relevant, for example "first polar body fragmentation lowers expected competence more strongly when maternal age ≥40." Items come from three sources: existing grading conventions, the mechanistic literature above translated into observable correlates, and a short open round in which panelists propose items.

We then recruit about 30 senior embryologists and reproductive endocrinologists and assign them at random to two panels of about 15. Both panels run three anonymous rounds with the usual median, interquartile-range and anonymized-rationale feedback. Consensus is a pre-registered rule: at least 75% of ratings in the top band and an interquartile range of at most 1. Items may also be classed "conditional consensus" in the HAH-Delphi sense. The outcome-anchored panel additionally receives, for every item, its age-stratified discrimination and calibration on a feedback cohort of clinical cycles, shown as a plain plot with confidence intervals. The classic panel receives a matched literature summary of the same length, so the arms differ only in whether the extra material is empirical outcome data. A third arm runs the same three rounds with a panel of several independently prompted language-model instances over the same literature corpus, following the HAH-Delphi design. It receives no outcome data.

The data come from a partner clinic with time-lapse recordings, maternal and paternal age, and outcomes: blastocyst formation for all embryos and euploidy where preimplantation testing was done. The data are split once, by patient and by date, into a feedback cohort that only the outcome-anchored panel ever sees and a validation cohort that nobody sees until the end. Where access permits, the public time-lapse set of Predicting Blastocyst Formation in IVF serves as an external check on the blastocyst endpoint, although it lacks age. Each panel's final consensus criteria are turned into a scoring rule in two forms. In the first, blinded embryologists who were not on any panel apply the rules by hand. In the second, a light model is trained to detect each criterion and the detections are combined with consensus weights.

Baselines

The naive reference is the clinic's current practice: the embryologist's routine grade, Gardner score for blastocysts, used as a single predictor of outcome. It shows how much any new criteria add over what is already done.

The strongest existing method for the blastocyst endpoint is the DINOv2 plus attention-LSTM predictor of Predicting Blastocyst Formation in IVF. Its authors report 96.4% accuracy and AUC 0.99 on their single-centre 704-video set. We retrain it on the union of feedback and training data available to our panels, with the same splits and a hyperparameter budget equal to that of our criterion-detector models. We report it overall and by age stratum. No reported age-stratified result was found on Lacuna for this model, or for any embryo model in the file. Matching it overall would not be a win for us. Our claim is won if consensus criteria, alone or added as inputs, give a confidence-interval-separated gain in AUC or calibration in the ≥38 stratum. Any gain must hold on the validation cohort.

The data-only baseline is an L1-regularized logistic regression with age interaction terms, fit on the feedback cohort over the same item bank, with tuning by cross-validation within that cohort. This asks whether experts add anything beyond an automated search over the same candidate criteria. The classic-Delphi arm is the ablation that isolates our key ingredient, the outcome feedback. The language-model arm tests whether the human panel is needed at all. For the consensus process itself, we compare against HAH-Delphi's reported 95% historical-item replication (HAH-Delphi). We do not treat that agreement figure as a success criterion here, because it measures agreement with past panels rather than with outcomes.

Why not just…?

Why not just train an end-to-end model on outcomes and skip experts? This is a real threat for the blastocyst endpoint, where a single-centre model already reports AUC 0.99. It is weaker where it matters. Euploidy and live birth in women over 40 are rare outcomes in small cohorts, and even 249-image blastocyst studies show unstable training and negative task interference (MEmEBG). The end-to-end baseline above settles it directly. If it beats every consensus arm in the ≥38 stratum on euploidy, we will report that the consensus criteria earn their keep only as interpretable, transferable rules, and we will test that claim separately on the external set.

Why not just aggregate many independent grades statistically instead of iterating? Pooling one-shot ratings from many graders reduces noise but cannot correct a bias that all experts share, such as overweighting polar-body extrusion in aged eggs, which the Sorbinil result suggests (Sorbitol accumulation). We include round-one ratings from both human arms, before any feedback, as this baseline. If round one matches round three on validation, iteration is worthless, and the experiment will show it.

Why not just fit the age interactions from data, as the regression baseline does? With about 60 items, multiple strata and few aged euploid outcomes, an unconstrained interaction search overfits. Experts restrict which interactions are plausible and can propose criteria absent from the bank. That defence is only a hypothesis. If the regularized regression matches the outcome-anchored panel on validation, the panel's contribution reduces to item generation, and we will say so.

Why not just use a language-model panel? HAH-Delphi shows high agreement with past panels, but its authors report shallow reasoning in the model (HAH-Delphi). The literature on aged-oocyte morphology is thin and partly contradictory: spindle failure versus cytoplasmic crowding (Artificially decreasing cortical tension). The language-model arm is scored on the same validation outcomes, so this question is answered rather than assumed.

Proposed Experiments and Decisive Checks

The primary endpoint is the difference in validation-cohort AUC for euploidy in the ≥38 stratum between rules from the outcome-anchored panel and rules from the classic panel. It is pre-registered with a bootstrap confidence interval. Secondary endpoints are blastocyst-formation AUC and calibration slope by stratum, and inter-rater agreement (Fleiss' kappa) when outside embryologists apply each arm's rules. The B/C-type grade confusion in Grading of mammalian cumulus oocyte complexes suggests that sharper criteria should raise kappa. We also measure the number of items whose consensus status differs between arms, and the cost in rounds and expert hours.

Three checks would be decisive. If the arms produce the same consensus items, outcome feedback does not move experts and the method fails at its first step. If the items differ but validation AUC does not, feedback moves opinion without improving accuracy, possibly by overfitting to the feedback cohort. The patient- and date-disjoint split exists to expose this. If the outcome-anchored rules win in older patients, and the effect survives the regression and end-to-end baselines, we will have shown that consensus can be pinned to biological outcomes. We will also have a reusable, age-conditional grading standard to replace subjective labels in the next generation of embryo models.