Expert consensus panels quietly make quantitative predictions. When a Delphi panel (a structured process in which experts rate items anonymously over several rounds, see the group's summary ratings, and revise) decides which items belong in a treatment-adherence questionnaire, it is betting that those behaviors matter for patients' futures. Those bets are almost never checked. Scale-development studies validate their instruments on one-time "snapshot" surveys, called cross-sectional surveys. These checks show whether the items hang together and whether a cutoff separates groups. They do not show whether the score predicts when a patient will reach the outcome the scale exists to prevent. We propose asking the panel for explicit numbers instead: for each adherence domain, how much should a one-standard-deviation improvement change the rate of progression to dialysis? Those numbers would then serve both as the prior in a survival model and as forecasts that can be scored against observed time-to-dialysis.
The opportunity is concrete. The 45-item Treatment Adherence Scale for non-dialysis chronic kidney disease (TAS-NCKD) was built through two Delphi rounds and validated on three cross-sectional hospital cohorts totaling 880 patients (Development and Validation of the Treatment Adherence Scale for Non-Dialysis Chronic Kidney Disease Patients in China). Its stated purpose is to delay end-stage renal disease, yet it was never linked to time-to-dialysis. The claim we will test has two parts. First, expert-elicited effect sizes, used as shrinkage targets in a Cox model, predict time-to-dialysis better than data-driven selection when events are scarce. Second, Delphi feedback rounds make experts more accurate, and not merely more alike. The second question has a clean answer once survival data exist, and no study in the file has asked it.
Core Claim
A proportional hazards model, or Cox model, describes each patient's instantaneous risk of an event as a shared baseline risk over time multiplied by $\exp(\beta^\top x)$. Here $x$ holds the patient's covariates and each $\exp(\beta_j)$ is a hazard ratio. Our proposal replaces the usual "shrink everything toward zero" penalty with shrinkage toward what the panel said. We estimate
$$\hat\beta = \arg\max_\beta \; \ell(\beta) - \tfrac{\lambda}{2}\sum_j \frac{(\beta_j - m_j)^2}{s_j^2},$$
where $\ell(\beta)$ is the Cox partial log-likelihood, $m_j$ is the panel's median elicited log hazard ratio for covariate $j$, $s_j$ is derived from the panel's interquartile range, and $\lambda$ is a single strength parameter chosen by cross-validation. The prior has two parts. The panel's median says where to shrink. The panel's disagreement says how hard to shrink: items the panel argued about are left to the data. The same $m_j$ and $s_j$ can also be scored directly as forecasts, by comparing them with the held-out estimates and their uncertainty, separately for Delphi round 1 and for the final round.
Relation to Prior Work
Three studies motivate the gap. Each pairs Delphi or structured expert consultation with a cross-sectional instrument and then stops at psychometrics. TAS-NCKD uses item-response models, confirmatory factor analysis, and a ROC cutoff of 179 points (TAS-NCKD). The mental health service questionnaire refines items through stakeholder consultation and then runs factor analysis (MHSDUQ). BRIDGE ranks workflow steps with a relative importance index over four Delphi sessions and never tests the roadmap (BRIDGE pilot study). We keep their instrument-building pipeline and add the one step they omit, an outcome over time.
The closest design linking a cross-sectional survey to a Cox model is NHANES with death-index follow-up. In that design, baseline survey and laboratory measures are fit with weighted Cox regression over a median of 91.2 months (RAR and mortality in NHANES). We borrow this "snapshot baseline plus linked follow-up" design. That study chose its predictors by convention, such as quartiles, and used no expert input, which is exactly the difference we test.
Two more papers supply methods and constraints. FastSurvival supplies a stable, fast solver for penalized Cox models (FastSurvival), and we reuse its surrogate-based coordinate descent with a recentered penalty. Its synthetic success, 100% support recovery of 15 true features among 1,200 correlated candidates, comes with many samples. Our regime has fewer than about 150 events, and that is the setting where priors could matter. A registry study found adjusted hazard ratios for kidney failure of 5.50 in intellectual disability, rising to 17.88 in under-20s (Koo et al.). It could only guess at behavioral explanations because billing codes contain no adherence measures. This motivates measuring adherence, and it warns that adherence effects may be heterogeneous.
One paper constrains the interpretation. A longitudinal dialysis study argues that snapshot measures of a dynamic quantity mislead (serum potassium and peritonitis). A single adherence survey is such a snapshot, so we treat it as a baseline exposure and do not make causal claims.
Research Program
The work has three components. The first is elicitation. We rerun a modified Delphi with nephrologists, renal nurses, and chronic-disease specialists recruited across more than one region, because BRIDGE's panel clustered geographically. Panelists give, for each of the five TAS-NCKD domains and for the strongest items, a median and a plausible range for the hazard ratio of dialysis initiation or a 40% eGFR decline per one-SD better adherence, adjusted for baseline eGFR and proteinuria. Answers from every round are archived. To limit fatigue, which BRIDGE reported in its later rounds, there are at most three rounds and the item set is kept short.
The second is the outcome cohort. We link the TAS-NCKD respondents from the Xi'an tertiary hospitals, and a fresh multicenter sample if linkage yields too few events, to hospital and dialysis records from the survey date onward. Follow-up starts at the survey. We adjust for kidney function at the time of the survey and run a 6-month landmark analysis to limit reverse causation, where sicker patients become more adherent.
The third is a simulation study. It maps when the expert prior helps and when it hurts, by sweeping prior bias and prior confidence across event counts. This frames the empirical result instead of leaving it as a single anecdote.
Baselines
The naive reference is the scale as published: the TAS-NCKD total score, or its 179-point cutoff, entered as a single covariate alongside clinical adjusters. If this is as good as anything else, the scale validates but the elicitation adds nothing. That would be a legitimate finding.
The clinical-only Cox model, using age, eGFR, proteinuria, diabetes, and blood pressure, gives the floor that adherence must improve on.
The strongest data-driven competitor is penalized Cox fit with FastSurvival on all 45 items plus clinical covariates, with the penalty tuned by the same nested cross-validation. Adaptive lasso runs alongside it. All methods get identical folds, an identical tuning grid size, and identical covariates. The only reported numbers in the file for this class of methods come from other data: 100% support recovery on FastSurvival's synthetic set, and C-index values of 0.816 for linear Cox and 0.867 for DeepSurv on METABRIC (DeepSurv). No reported result was found on Lacuna for any survival model on a CKD adherence cohort, so the bar is set by our own run. Beating it means a higher held-out C-index and a better integrated Brier score, with a bootstrap interval that excludes zero difference. With this sample size, we expect any gain to be a few hundredths of C-index at most.
Three ablations isolate the key ingredient. The first uses CVI-style relevance weights only, applying item-specific shrinkage strength without a recentered target. This tests whether quantitative elicitation adds anything beyond the relevance ratings Delphi studies already collect. The second replaces the panel's $m_j$ with permuted values. The third compares round-1 priors with final-round priors.
Why not just…?
Why not just use lasso or FastSurvival and skip the panel? With 45 correlated items and roughly 100 events, data-driven selection is unstable, and FastSurvival's recovery guarantee was shown in a sample-rich synthetic setting. The comparison is built into the design. If FastSurvival matches the expert-centered model on held-out C-index and calibration, we report that the panel's numbers carry no predictive information beyond the data. The simulation will show at what event count the prior stops mattering.
Why not just validate the total score in a Cox model? This is the real threat. A plain validation study may capture most of the clinical value. We run it as the naive baseline. The elicitation is justified only if domain-level structure or the forecast-scoring result adds something: either improved prediction, or evidence about whether Delphi feedback improves accuracy. The second contribution survives even if the first fails.
Why not take priors from published hazard ratios instead of experts? In the file, no study reports domain-level adherence hazard ratios for CKD progression. The registry evidence (Koo et al.) concerns diagnoses, not behaviors. Where any literature value exists for a covariate, such as overall medication adherence, we will add a literature-centered prior as a further comparator.
Why not convene a face-to-face committee? Anonymity is the Delphi's defense against dominant voices (ankle cartilage consensus). Our design is the first that could measure whether that defense yields more accurate forecasts. The round-1 versus final-round comparison is a partial test of it.
Decisive Checks
Three results decide the claim. First, on held-out folds, the expert-centered Cox model beats tuned FastSurvival and the total-score model on C-index and integrated Brier score. The permuted-prior ablation must erase that gain, which shows the gain comes from the experts' specific numbers and not from shrinkage itself. Second, the panel's final-round 50% intervals contain the empirical log hazard ratios about half the time, and final-round medians sit closer to the estimates than round-1 medians. If they sit farther, Delphi convergence is producing conformity, and that would be a publishable negative result. Third, items with high panel disagreement show wider empirical uncertainty, which tests whether panel disagreement reflects real uncertainty. If all three fail, the TAS-NCKD scale still receives its first outcome validation. The simulation also gives future Delphi-to-model pipelines a quantitative rule for when expert numbers are worth eliciting.
References
- Development and Validation of the Treatment Adherence Scale for Non-Dialysis Chronic Kidney Disease Patients in China: A Mixed-Methods Study
- Development and validation of the mental health service demand and utilization questionnaire
- BRIDGE pilot study: a bilateral regulatory investigation of data governance and exchange
- Relationship between the Red Blood Cell Distribution Width-to-Albumin Ratio and the Risk of Chest Pain and Mortality among U.S. Adults: Evidence from the NHANES 2007–2016
- FastSurvival: Hidden Computational Blessings in Training Cox Proportional Hazards Models
- Association between intellectual disability and autism spectrum disorder with kidney failure
- Association between serum potassium levels and peritonitis in peritoneal dialysis patients: a longitudinal study
- DeepSurv: personalized treatment recommender system using a Cox proportional hazards deep neural network
- Conservative Management and Biological Treatment Strategies: Proceedings of the International Consensus Meeting on Cartilage Repair of the Ankle