Large genetic studies pool patients from hospitals on different continents. Before pooling, experts have to agree on who counts as a "case" of a disease. Reviewing ambiguous charts and settling disagreements between sites takes clinician time. When that review needs a live meeting, it also takes meeting slots that fall inside everyone's working hours. Those slots are scarce, and in practice they favor whichever regions overlap most comfortably. This note argues that the choice of which sites' cases get adjudicated is a statistical decision, not just a logistical one. A polygenic score is a single number summing a person's many small genetic risk variants. If the disease labels are cleaned unevenly across sites, the label errors differ by site. Site is tied to ancestry, and ancestry is tied to the distribution of the score, so the errors end up correlated with the very score being studied. That can bias a score–disease association in either direction, not merely shrink it toward zero, and post-hoc score-weighting methods cannot repair it.
We propose to treat cross-site phenotype adjudication as a two-phase measurement design. The adjudicated cases do two jobs: they fix labels, and they calibrate a site-specific error model that corrects the downstream association. Adjudication effort, including scarce synchronous deliberation slots, would be allocated to minimize the variance of the corrected estimate, subject to real time-zone feasibility. The two-sentence version is this. Uneven, time-zone-driven expert review across biobank sites creates label errors that are correlated with genetic scores, and that is worse than ordinary noise. Allocating review to equalize and minimize that error is a solvable design problem whose objective comes from measurement-error statistics rather than from fairness preferences.
Core Claim
The polygenic-score literature already shows that label errors in the training or target cohort degrade score performance. Evaluation of Phenotyping Errors on Polygenic Risk Score Predictions simulated misclassification on eMERGE and found two things. Random errors degrade accuracy uniformly, with AUC falling to about 0.5 at 50% sensitivity and specificity. Errors that depend on covariates or related diagnoses cause severe and unpredictable degradation. That paper motivates this proposal but treats the error rates as given. It does not ask what process produces differential error in a multi-site consortium, or how to control that process.
The deliberation literature supplies the missing process. Scheduling Virtual Conferences Fairly shows that schedules optimized only for total attendance systematically exclude people in unfavorable time zones. The ISMAR 2019 program committee (Gabbard et al.) connected Beijing, Blacksburg and Coburg for two days. It cleared consensus cases in bulk and reserved live debate for 68 borderline papers, which accepts off-hour fatigue for at least one region. Our claim is that the same pattern, applied to disease-label adjudication, generates exactly the site-differential error Li et al. found most damaging.
The mechanism fits in one line. Let $Y$ be the true binary disease status, $Y^*$ the recorded label, $G$ the polygenic score, and $s$ the site. Let $\mathrm{Se}_s$ and $\mathrm{Sp}_s$ be the site's sensitivity and specificity after whatever adjudication the site received. Under a linear probability model,
$$ E[Y^* \mid G, s] = (1-\mathrm{Sp}_s) + \lambda_s\, E[Y \mid G, s], \qquad \lambda_s = \mathrm{Se}_s + \mathrm{Sp}_s - 1 . $$
This follows from averaging the misclassification probabilities over the true status. Within a site, the score's slope is shrunk by the factor $\lambda_s$, which is ordinary attenuation. Pooled across sites, two further distortions appear. The intercept term $(1-\mathrm{Sp}_s)$ shifts apparent prevalence by site. Because mean $G$ differs across ancestry-distinct sites, that shift loads onto the score slope and can inflate or reverse it. Separately, heterogeneous $\lambda_s$ makes the pooled slope a weighted mixture that no single correction factor recovers. Adjudication raises $\lambda_s$ and narrows the uncertainty in it, but only at the sites whose cases are actually reviewed.
Research Program
The first component is estimation. We fit a site-specific misclassification model, with sensitivity and specificity per site and optionally per stratum of related diagnoses, using the adjudicated subsample as a validation set. The PGS–disease coefficient is then corrected by likelihood-based misclassification adjustment. The individual-level analysis can instead run in decentralized form: Decentralized Mixed Effects Modeling in COINSTAC fits multi-site mixed models by exchanging only product matrices and aggregate estimates. We reuse that computational pattern and add the error model to it. COINSTAC itself addresses neither polygenic scores nor label error.
The second component is design. Each candidate case has a posterior probability of being mislabeled, estimated from the site's automated phenotype algorithm and the patient's related diagnoses. Adjudication comes in two kinds. Asynchronous single-reviewer chart review is cheap. Synchronous cross-site deliberation is needed when sites disagree on how a definition applies, and it is costly: it needs a slot inside the working hours of the involved sites' experts.
We choose which cases get which kind of review in order to minimize the estimated variance of the corrected coefficient. Its dominant terms scale with the variance of each $\hat\lambda_s$ weighted by that site's share of score variation. Slots are assigned using the integer-programming machinery of mFairConf (Patro et al.), namely relaxations with repeated rounding of fractional solutions. The objective is different, though. Patro et al. balance audience size against equity in satisfaction. We minimize the statistical error of a downstream estimate. The site-coverage constraint then emerges from that variance objective instead of being imposed as a fairness weight. When the two diverge, for example when statistical efficiency wants more slots for a small, error-prone site than equity would give, we report it.
The third component is a small human study to check the premise that off-hour deliberation is worse. Clinicians at three time-zone-separated sites would adjudicate a common set of ambiguous charts in slots at varying local times. We measure agreement with a reference panel as a function of local hour. If agreement does not fall off-hours, the fatigue term drops out of the design and only coverage matters. That simplifies the method without invalidating it.
Baselines
The naive reference point is to use the automated site-specific phenotype labels with no adjudication and fit a pooled PGS regression with site fixed effects. It shows how large the bias is when nothing is done. Li et al. report AUC dropping to about 0.5 at 50% sensitivity and specificity under random error. That is the only reported degradation figure in this setting, and we will reproduce it as a sanity check of our simulator before adding site structure.
The strongest statistical competitor is a site-stratified random validation subsample of the same total size, followed by the same misclassification correction. This is textbook two-phase design, and it is the method most likely to match ours. It removes the coverage problem simply by sampling every site. It will be run with an identical adjudication budget and identical correction code. The only difference is that its cases are chosen at random within sites rather than by our variance-targeted rule under time-zone constraints. Beating it means a smaller mean squared error of the corrected coefficient at equal budget, with confidence-interval coverage at least as good. No reported result was found on Lacuna for this design in a PGS setting.
The strongest existing scoring-side competitor is robust summary-statistic scoring. Identifying and correcting for misspecifications in GWAS summary statistics introduced LDpred2-auto-rob, LDpred2-low-h2 and lassosum2. Their paper reports LDpred2-low-h2 improving squared correlation from 0.134 to 0.164 under sample-size misspecification. It also reports lassosum2 falling from 0.173 to 0.166 with a mismatched reference panel. We run these on the unadjudicated labels with default tuning grids. The prediction is that they recover weight-side error but leave the pooled association biased, because the error lies in the target labels.
For calibration of what correct coverage looks like, we cite PENGUIN (Zhao et al.). It fits all SNPs jointly to remove residual genetic confounding and maintained 95% confidence-interval coverage in WTCCC-based simulations. It addresses a different error, confounding in exposure–outcome studies, so it serves as a coverage standard rather than a head-to-head competitor.
Two ablations isolate the key ingredient. The first uses our targeted rule with an "efficiency-only" schedule, in the style of the ISMAR fixed hubs, that ignores whether distant sites can attend. It shows the cost of uneven coverage. The second is an oracle that ignores time-zone feasibility entirely. It bounds how much of the loss scheduling can ever recover.
Why not just…?
Why not just use asynchronous Delphi rounds or fixed phecode dictionaries? This is a real threat. If asynchronous single-reviewer adjudication achieves the same site-level sensitivity and specificity, synchronous deliberation is unnecessary, and our method reduces to allocating chart reviews. The design includes an asynchronous-only arm for exactly this reason. The statistical core, correcting and equalizing $\lambda_s$ across sites, holds either way. The evidence that asynchronous coordination loses effort comes from other domains: TeachNow found that spontaneous real-time help beat scheduled or forum help, and Murphy et al. documented asynchronous breakdown in distance teams. Neither concerns clinical adjudication, so we treat the question as open. A fixed dictionary applied identically everywhere still yields different error rates when coding practices differ across hospitals. Uniform rules do not imply uniform $\lambda_s$.
Why not just take a stratified random validation sample? It may be nearly as good, and we say so. Its weakness is that it spends reviews on cases that are almost certainly labeled correctly. The ISMAR bulk-accept logic implies that most information lies in the borderline cases. The head-to-head comparison at matched budget settles the question. If the gain is small, the practical recommendation becomes "stratify by site," which is still a useful and currently unstated rule for consortia.
Why not just apply better score shrinkage? LDpred2 and lassosum2 correct misestimated variant weights. The bias here sits in the outcome labels of the target analysis and in their correlation with site. Li et al. found that downstream scoring does not undo differential phenotyping error. Our shrinkage baseline tests this directly on site-structured error.
Why not just meta-analyze sites with random effects? Random effects absorb differences in site means but do not undo per-site attenuation, so averaging attenuated slopes yields an attenuated average. Fu et al. also show that standard meta-analysis struggles when studies use different working models. We include site-level random-effects meta-analysis on the unadjudicated labels as a further comparison.
Proposed Experiments
The main testbed is a simulation on real eMERGE genotypes (dbGaP phs000888.v1.p1), following the Li et al. protocol. It reuses real linkage structure and a disease architecture with shared causal variants between the target and a related diagnosis. We partition individuals into synthetic sites whose mean PGS differs, as it would across ancestry groups. Each site receives its own sensitivity and specificity, with some sites made more error-prone to mimic less-adjudicated centers.
Adjudication capacity is simulated from real time-zone offsets. Experts are available during local working hours, and an optional accuracy penalty applies to off-hour slots, calibrated by the human pilot. We vary the budget, the spread of error rates across sites, and the correlation between site and mean score. We report bias, mean squared error and interval coverage for the PGS coefficient, along with AUC.
The decisive check is whether the efficiency-only schedule produces a pooled coefficient biased away from zero, not merely toward it, when site-level specificity differences align with mean-score differences. The first equation predicts this, and the ISMAR-style design would suffer from it. If that sign-flip or inflation does not appear at realistic error spreads, the central concern of this proposal is overstated. The contribution would then shrink to efficient validation-sample design.
The second decisive check is whether our allocation beats stratified random validation at matched budget. The third is whether the corrected estimate recovers within-site slopes when run through the decentralized COINSTAC-style computation with no pooled individual data. If the InterLymph subtype definitions used by Fu et al. could be accessed with site identifiers, a retrospective analysis would test whether site-level label disagreement predicts heterogeneity in the PGS–subtype associations. We list this as contingent, since the file documents no adjudication records for that consortium.
References
- Evaluation of Phenotyping Errors on Polygenic Risk Score Predictions
- Scheduling Virtual Conferences Fairly: Achieving Equitable Participant and Speaker Satisfaction
- Message from the ISMAR 2019 Science and Technology Program Chairs and TVCG Guest Editors
- Decentralized Mixed Effects Modeling in COINSTAC
- Identifying and correcting for misspecifications in GWAS summary statistics and polygenic scores
- Controlling for polygenic genetic confounding in epidemiologic association studies
- TeachNow: Enabling Teachers to Provide Spontaneous, Realtime 1: 1 Help in Massive Online Courses
- Experiences in Teaching eXtreme Programming in a Distance Learning Program
- Improve the model of disease subtype heterogeneity by leveraging external summary data