Jointly Calibrated Ensembles of Epidemiological Designs for Drug and Vaccine Safety

Observational health databases (insurance claims and electronic health records covering millions of patients) are used to ask whether a drug or vaccine causes a harmful outcome. Each standard study design for this question has its own blind spot. A cohort study compares people who took the drug with similar people who did not, so it is fooled by differences between those groups that the records do not capture. A self-controlled design compares each patient with themselves at different times, so it is fooled when the outcome itself changes who keeps taking the drug or who survives. Ensemble methods from other fields, such as combining several particle-picking programs in cryo-electron microscopy, rest on the idea that independent methods make different mistakes while real signals show up in all of them. This proposal applies that idea to safety studies, with one change: we never assume the designs' mistakes are independent. We measure how they are correlated.

The measurement uses negative controls, which are drug-outcome pairs known to have no causal link. Running every design on hundreds of these pairs shows how far each design is off and whether different designs tend to be off in the same direction for the same pair. That estimated joint error distribution gives a combined estimate of the true effect, with confidence intervals that account for shared bias. It also gives a formal test for when the designs disagree more than their known error patterns allow. A two-sentence summary: run several epidemiological designs in parallel, learn the covariance of their systematic errors from negative controls, and combine them by weighting each design according to how much independent information it actually carries. The goal is better sensitivity than the best single design at the same calibrated false-positive rate, plus a discordance flag that tells an analyst when a design's assumptions have probably broken.

Core Claim

The closest prior work shows that combining designs the obvious way fails. Best of intent, worst of both worlds tested a serial pipeline in four databases: a historical-comparator screen, then a self-controlled case series to confirm what the screen flagged. Type II error exceeded 50% in nearly all analyses, and even after calibration the serial approach had the lowest sensitivity of any approach tested, because every true signal the noisy first stage missed was never examined again. This result motivates our proposal and also limits it. Gatekeeping is ruled out, and so is any scheme that throws evidence away at an early stage. The OHDSI benchmark (How Confident Are We About Observational Findings in Health Care) and the vaccine design comparison (Vaccine Safety Surveillance Using Routinely Collected Healthcare Data) supply the method we reuse, empirical calibration with negative and synthetic positive controls. Both apply it to one design at a time and treat designs as competitors. The second paper notes that systematic error differs sharply across designs but never uses that fact.

Our claim is that a parallel combination, calibrated jointly rather than design by design, recovers the sensitivity that serial gatekeeping loses and keeps calibrated false-positive control. The specific new ingredient is an estimate of the off-diagonal error structure, meaning how the designs' errors co-vary. Pharmacovigilance consensus filtering on spontaneous reports, such as requiring agreement across four disproportionality statistics (reporting odds ratio, proportional reporting ratio, and the two Bayesian shrinkage measures IC and EBGM, as in the tiotropium FAERS study), shows the failure we want to avoid. All four statistics are computed from the same contingency table, so their agreement is close to automatic and tells us little about whether the signal is real.

Research Program

For each drug-outcome pair $i$ we run $D$ designs on the same database. The designs are the five from the OHDSI benchmark: a new-user cohort with large-scale propensity scores, a self-controlled cohort, case-control, case-crossover, and a self-controlled case series. We add the index-date variants of the cohort design that the vaccine study found to be fragile. Each design produces a log effect estimate, and we collect these into a vector $\hat\theta_i \in \mathbb{R}^D$. We model this vector as

$$\hat\theta_i \sim \mathcal{N}\big(\beta_i \mathbf{1} + \mu(\beta_i),\; \Sigma + S_i\big),$$

where the symbols mean the following. $\beta_i$ is the true log effect. $\mathbf{1}$ is a vector of ones. $\mu(\beta)$ is the mean systematic error of each design, which may depend on the true effect size and is fitted with the synthetic positive controls. $\Sigma$ is the $D\times D$ covariance of systematic error across designs, fitted on negative controls. $S_i$ is the sampling covariance for pair $i$.

The sampling covariance is not diagonal, because the designs reuse the same cases. We therefore estimate the cross-design entries of $S_i$ by a patient-level bootstrap.

Given $\Sigma$, $\mu$, and $S_i$, the combined estimate of $\beta_i$ is a generalized least-squares estimate, which is a weighted average whose weights fall automatically on designs whose errors are redundant with others. Its calibrated interval follows from the same model. The model also yields a discordance statistic: the squared Mahalanobis distance of the residual $\hat\theta_i - \hat\beta_i\mathbf{1} - \mu(\hat\beta_i)$, which follows a chi-squared distribution with $D-1$ degrees of freedom under the model. When this statistic is large, the designs disagree more than their learned error structure allows. In that case we report which design's residual drives the disagreement rather than forcing a single verdict.

The project has three parts. The first is software that runs all designs in parallel on OMOP-format databases (OMOP is a shared data format) and fits the joint model with leave-one-out cross-validation over controls, so that no control is used to calibrate itself. The second is an empirical study of $\Sigma$: how large the off-diagonal correlations are, whether they hold steady across databases, and which design pairs come closest to independent. The third is an evaluation against the baselines below. We treat spontaneous-report statistics from the FDA Adverse Event Reporting System (FAERS) only as an optional extra member whose added value the fitted model would show. FAERS is not the core of the proposal, because it lacks denominators and drug-exposure timing, so it cannot be calibrated on the same negative controls in the same way.

Baselines

The naive reference point is a single uncalibrated new-user cohort with large-scale propensity scores, the default design in Drawing Reproducible Conclusions from Observational Clinical Data with OHDSI. The OHDSI benchmark reports that this design's 95% confidence interval coverage in CCAE (a US commercial claims database) is 0.73 uncalibrated and 0.89 calibrated, and that uncalibrated Type I error across methods reaches 82%.

The strongest existing competitor is the calibrated self-controlled case series that adjusts for all other drug exposures. In the OHDSI benchmark it reached an area under the ROC curve of 0.95 to 0.98 across CCAE, MDCR, PanTher/Optum, and JMDC. The vaccine study found it the fastest design at detecting weak (1.5x) effects after calibration. For this design's sensitivity at a fixed calibrated 5% Type I error, I found no reported result on Lacuna. We will compute it ourselves with the published OHDSI pipeline on the same controls, databases, and calibration procedure as the ensemble. Beating it means higher sensitivity at matched calibrated 5% Type I error, with coverage at least as close to 95%, on the pooled controls and in at least three of four databases. Because its discrimination is already near ceiling, we will not claim a win on the area under the ROC curve alone.

We also include the serial historical-comparator-then-case-series pipeline from Best of intent, worst of both worlds, reproduced on its vaccine benchmark of 93 negative and 279 imputed positive controls. Its Type II error above 50% is the figure to beat. A strict all-designs-must-agree vote, the epidemiological analogue of the four-statistic FAERS rule, serves as a further baseline.

The ablation that isolates our key ingredient keeps everything identical except that $\Sigma$ is forced to be diagonal. This amounts to calibrating each design separately and then pooling by inverse variance. If the full model does not beat this ablation, the covariance idea adds nothing. All methods use the same controls, cross-validation folds, and databases. None gets extra tuning: the ensemble's only fitted quantities are $\mu$ and $\Sigma$, fitted on held-out folds.

Why not just…?

Why not just use the calibrated self-controlled case series alone? This is the real threat, given its 0.95 to 0.98 benchmark discrimination. Its weaknesses are structural. It is biased when an outcome shortens follow-up through death or causes the drug to be stopped. It suits poorly defined continuous exposures badly. Best of intent, worst of both worlds reports that its performance varied across databases. Benchmark controls may under-sample exactly these failure cases, so we will evaluate on two subsets. One is controls whose outcomes carry high short-term mortality. The other is semi-synthetic controls with event-dependent exposure stopping injected. If the case series alone matches the ensemble on both the full benchmark and these subsets, we will report that the ensemble is unnecessary.

Why not just calibrate each design separately and average them, or take the smallest p-value? That is the diagonal-$\Sigma$ ablation, plus a minimum-p-value variant with Bonferroni correction. If cohort and self-controlled errors turn out nearly uncorrelated on negative controls, this simpler pooling is enough and our extra machinery is not justified. Measuring $\Sigma$ settles the question directly. The serial study's finding that separately calibrated designs still combined poorly suggests, but does not prove, that shared error matters.

Why not just screen and then validate? Best of intent, worst of both worlds has already shown that sensitivity collapses under this approach. We reproduce it only as a reference point. Screening on FAERS has the extra problem that its statistics are confounded by the drug's indication: the tiotropium-dyspnea reporting odds ratio of 162.04 reflects COPD symptoms more than drug harm.

Why not just adjust for more covariates in one cohort? Large-scale propensity adjustment still left coverage at 0.73 before calibration in CCAE. The vaccine study showed that a single choice of comparator index date can flip a cohort result between false positive and false negative. Adding covariates cannot fix a wrong structural choice of this kind, whereas a parallel self-controlled design is not exposed to it.

Proposed Experiments

The primary experiment reruns the OHDSI Methods Benchmark: 200 negative controls and 600 synthetic positive controls at effect sizes of 1.5, 2, and 4, in CCAE, MDCR, Optum/PanTher, and JMDC. We report coverage, calibrated Type I error, sensitivity at 5%, and mean squared error of the combined log effect. The second experiment repeats the analysis on the vaccine surveillance benchmark (CCAE, MDCR, MDCD, and Optum EHR, across six vaccine cohorts) with sequential monthly looks, and adds time to detection. The third experiment tests transfer: we fit $\Sigma$ in one database and apply it in another, to see whether error structure is a property of the designs or of the data.

The decisive checks are three. First, the estimated off-diagonal error correlations must be clearly nonzero yet below one. If they are near zero, the simple pooled ablation wins. If they are near one, no ensemble helps. Second, the full model must beat both the diagonal ablation and the calibrated case series on sensitivity at matched calibrated error. Third, the discordance statistic must flag the injected assumption-violation controls more often than ordinary controls. One caveat limits every conclusion. Synthetic positive controls are created by injecting outcomes into negative-control data, so they inherit the negative controls' confounding but may not reproduce the timing patterns of real adverse effects. We will therefore also report results on the subset of real, literature-established positive controls that the benchmark databases support.