Conformal Risk Sets for Drug Cocktails Under Order Shift and Memorization

Many patients take five or more drugs at once, and some of the most dangerous adverse reactions appear only when a third or fourth drug is added to a pair that is harmless by itself. Machine-learning models of drug interactions are increasingly asked to flag such cocktails. A safety tool needs more than a score. It needs a statement like "with 90% probability the true outcome is in this set," which is exactly what conformal prediction provides. Conformal prediction is a wrapper that turns any model's scores into prediction sets with a guaranteed coverage rate, as long as the calibration examples and the future cases are statistically interchangeable. In polypharmacy that condition fails in two ways. First, calibration data are dominated by well-documented drug pairs, while deployment concerns larger, newer cocktails. Second, the strongest predictors are now text-pretrained language models that may have read the very interaction databases used to calibrate them. Memorized calibration examples look easy, so the conformal threshold comes out too small, and the coverage promise silently breaks on combinations the model has never seen.

This note proposes to measure and repair that failure. We will build a time-split polypharmacy evaluation in which the test cocktails were first reported after the models' pretraining cutoffs. We will show how the coverage of standard conformal sets degrades along two axes, cocktail size and model familiarity. We will then test a stratified calibration scheme that conditions on a few cheap, combination-level shift features instead of estimating a density over the joint space of drugs. In two sentences: conformal guarantees for drug cocktails are only as good as the match between calibration and deployment, and that match is broken by both cocktail size and training-data memorization. We quantify both breaks on a contamination-free benchmark and fix them by calibrating within strata defined by order, drug novelty, and textual familiarity.

Core Claim

The obvious combination of these directions, "apply conformal prediction to higher-order drug interactions," is not done anywhere in the file. It is also nearly trivial on its own. Conformal prediction is model-agnostic, so wrapping the arbitrary-order graph-matching classifier of Chiang et al. in a split-conformal layer is an afternoon's work, and it carries the usual guarantee when the data are interchangeable. The open problem is what happens when they are not. Our claim has two parts.

The first is empirical. On cocktails of order $k \ge 3$ that involve newly marketed drugs or were first reported after a model's pretraining cutoff, pooled conformal sets calibrated on historical data undercover. The size of the shortfall grows with cocktail order, with the number of drugs absent from calibration, and with how familiar the calibration examples were to the model. The last effect is the new one. Memorization lowers calibration nonconformity scores, and that biases the threshold downward.

The second part is methodological. Partitioned ("Mondrian") calibration fixes most of this gap at a fraction of the cost of density-ratio weighting. Each test cocktail is calibrated only against past cocktails that share its stratum, defined by order bucket, number of novel drugs, and a familiarity score. Within each stratum, coverage then holds exactly under within-stratum exchangeability.

Concretely, for a stratum $g$ with calibration scores $s_1,\dots,s_{n_g}$, the threshold is the $\lceil (n_g+1)(1-\alpha)\rceil / n_g$ empirical quantile of those scores. The prediction set for a test cocktail $x$ in stratum $g$ is

$$C(x) = \{\, y : S(x,y) \le \hat q_g \,\},$$

where $S$ is the nonconformity score and $1-\alpha$ is the target coverage. This is standard machinery. What is new is the choice of strata, the familiarity signal, and the benchmark that makes the failure visible.

Positioning Against Prior Work

The work that motivates this proposal comes from the shift and benchmark literature. How Reliable is Your Regression Model's Uncertainty Under Real-World Distribution Shifts? shows that conformal and non-conformal intervals alike fall from 90% to 55–70% coverage under real shifts. Benchmarking Graph Learning for Drug-Drug Interaction Prediction shows that drug-pair models evaluated on random splits overstate performance: EmerGNN's F1 falls from 56.9% to 34.0% when the test drug is structurally new. The same paper also reports that the text-based DDI-GPT is the most robust model, and it attributes this to pharmacological pretraining without checking for leakage. That untested attribution is our entry point.

The contamination literature supplies the evaluation design. Unveiling Causal Reasoning in Large Language Models: Reality or Mirage? builds a test set from events after the training cutoff. DyVal generates tasks procedurally so they cannot have been memorized. Studying and improving reasoning in humans and machines uses token-level surprise to detect memorized strings. We borrow the post-cutoff idea and the surprise measurement, and we apply both to the conformal calibration set rather than only to the test set.

The conformal methods we build on, and must beat or match, are these. CoDrug reweights calibration scores by a kernel density estimate of the test-to-calibration ratio for single molecules. It explicitly struggles with small, imbalanced calibration sets and does not address drug combinations. Conformal Prediction for Uncertainty Estimation in Drug-Target Interaction Prediction groups entities by their residual distributions (CCP-NC) to get subgroup coverage, but only for single drug–protein pairs. Act or Defer stratifies conformal selection by class so that rare positive classes keep their error budget. Relative to all three, our strata are defined by the combinatorial shift itself and by memorization, and our test data are deliberately unseen by the models.

Higher-order interaction modeling plays the role of the application regime, not the contribution. The review of machine-learning DDI approaches and Chiang et al. argue that pairwise models miss cocktail effects. I2Mole and GenRel-DDI are strong pairwise predictors that their authors concede cannot handle three or more drugs. We use them as pair-based base models that are aggregated over a cocktail, and we do not propose a new interaction architecture.

Research Program

The first component is a contamination-controlled polypharmacy benchmark. We follow Chiang et al.'s construction from FDA adverse event reports (FAERS). Each report's co-administered drug set is a combination of arbitrary order, and the label is the presence of a target adverse reaction. We start with myopathy so the setting matches theirs, then add a handful of other high-severity reactions.

We split by report date. Combinations first reported before a reference date form the training and calibration pools. Test combinations are those first reported after the pretraining cutoffs of the language models we evaluate. We further tag each test cocktail by its order $k$, by how many of its drugs are absent from the calibration pool, and by whether its constituent pairs appear in DrugBank or TWOSIDES.

Spontaneous reports carry real limits: under-reporting, Weber effects, and no verified negatives. Coverage is therefore measured against reported outcomes, and we state this plainly. As a sanity check we also run on the pairwise DDI-Ben S1 and S2 splits, where the labels are curated.

The second component is a familiarity score for each combination. For text-based predictors, we compute the language model's per-token surprise on a templated description of the cocktail and its constituent pair monographs. We also record a documentation indicator: whether any constituent pair has a public interaction record that predates the model's cutoff. For graph and kernel predictors that have no text pretraining, familiarity reduces to training-set overlap of the constituent pairs.

The third component is stratified calibration. We use coarse strata, crossing order bucket ($k=2$, $3$–$4$, $\ge 5$) with zero versus at least one novel drug and with low versus high familiarity. Strata with too few calibration points are merged by a rule fixed before any test data are seen. Within each stratum we calibrate class-conditionally, so the rare adverse class keeps its own coverage target, following the logic of Act or Defer.

We will not claim guarantees for orders larger than any seen in calibration. For those cocktails we report the empirical coverage and use the most conservative stratum threshold among those available.

Baselines

The naive reference point is the base predictor's own probability, thresholded at 0.5 or at the level that matches the target error rate on validation, with no conformal layer at all. This shows how miscalibrated the raw scores are on new cocktails.

The second reference is pooled split conformal prediction. One threshold is computed from all calibration cocktails and applied to every test cocktail. This is the default practice that our claim says breaks, so it is the most direct comparison.

The strongest existing competitor is CoDrug-style weighted conformal prediction (CoDrug), adapted to cocktails by estimating density ratios with a kernel density estimate on a pooled cocktail embedding. It is the strongest because it is the only cited method designed specifically to restore coverage under covariate shift in drug data. On AMES with fingerprint shift, CoDrug reports that standard conformal prediction reached only 34% coverage at a 90% target, while CoDrug restored coverage close to 90 percent. It also reports coverage-gap reductions of 35% on average and 60% under de novo design shift.

CCP-NC from the drug-target conformal paper is the strongest subgroup method. It clusters by residuals instead of by our shift features, and we include it with its cluster count tuned on validation data. The file gives no numeric interval widths for CCP-NC, and there is no reported result found on Lacuna for any conformal method on higher-order polypharmacy.

All conformal variants wrap the same three base predictors on identical calibration and test cocktails, with the same calibration-set size and an equal hyperparameter search budget, which covers bandwidth for CoDrug, cluster count for CCP-NC, and merge threshold for ours. The three base predictors are:

  1. Chiang et al.'s graph-matching kernel support vector machine, which reports AUC 0.912 and accuracy 83.6% on myopathy, against 76.5% for a probabilistic kernel. It is the strongest arbitrary-order predictor in the file.
  2. EmerGNN or GenRel-DDI trained on pairs and aggregated by taking the maximum risk over all pairs in the cocktail.
  3. A DDI-GPT-style text predictor.

Two ablations isolate the key ingredients. The first drops the familiarity axis, leaving order and novelty only, to test whether memorization stratification adds anything. The second drops the order axis.

We count as beating a baseline only the following. Our method must hold worst-stratum class-conditional coverage within two points of target, while pooled conformal prediction misses by more. It must also hold that coverage with average set size no larger than CoDrug's at matched coverage. If CoDrug matches us on both, our method contributes no improvement.

Why not just…?

Why not just decompose the cocktail into pairs and use a strong pairwise model? Because conformal prediction is model-agnostic, this changes the efficiency of the prediction sets, not their validity. We therefore include it as a base predictor rather than as a rival method. The evidence that it loses information comes from Chiang et al., where order-2 kernels over the whole cocktail beat order-1 representations. It also comes from the review's argument that a third drug can change the risk of a pair. Our experiments test this directly: if the max-over-pairs predictor yields set sizes as small as the cocktail-level kernel on $k \ge 3$ strata, then higher-order modeling buys nothing here, and we will report that.

Why not just use CoDrug's density weighting on a cocktail embedding? This is the real threat. Weighted conformal prediction is the principled answer to covariate shift, and if its density ratios are accurate it should handle order and novelty shift implicitly. CoDrug itself reports trouble with a calibration set of only 19 positive samples on ClinTox, and adverse cocktails of order five or more will be far sparser than that. Density-ratio weighting also cannot see memorization, because a memorized calibration example and a fresh one can have identical embeddings but different score distributions. The head-to-head comparison and the injected-memorization test below settle the question.

Why not just drop the language model and use predictors without text pretraining? That removes the contamination question, but it throws away the model that DDI-Ben found most robust on rare interaction types. Whether that robustness is real or memorized is exactly what we measure. If the text predictor's advantage disappears on post-cutoff cocktails, dropping it is the right recommendation and this proposal will say so. If its advantage survives, stratified calibration lets practitioners keep it safely.

Why not just ensembles or uncalibrated quantile estimates? The real-world regression shift study found that ensembles fall to about 59% coverage on Cells-Tails and that all methods fall below 90% on every shifted real dataset. We include a deep ensemble over the neural base model as a supplementary point, but we do not expect it to compete on worst-stratum coverage.

Proposed Experiments

The first decisive check is a controlled memorization test that does not depend on guessing what a pretraining corpus contains. We fine-tune the text predictor for a few epochs on half of the calibration cocktails, leaving the other half and the test set untouched. We then compare conformal thresholds computed from the memorized half against those computed from the clean half. The prediction is a lower threshold and undercoverage on the post-cutoff test cocktails when calibration uses the memorized half. Familiarity stratification should remove the gap. If no gap appears, the memorization arm of the proposal is wrong, and the work reduces to order and novelty stratification.

The second check is the main benchmark result. We report class-conditional coverage of the adverse class and average set size for every method crossed with every base predictor, broken down by order bucket, novelty, and familiarity. These are reported on the time-split FAERS cocktails at $\alpha = 0.1$ and $\alpha = 0.05$, alongside DDI-Ben S1 and S2 for pairs. The success criterion is the one stated under Baselines. We also plot the coverage shortfall of pooled conformal prediction against order $k$ and against mean calibration familiarity, to test the claimed monotone trends.

The third check probes extrapolation honestly. We calibrate only on cocktails with $k \le 4$ and test on cocktails with $k \ge 5$. We report how far each method's empirical coverage falls below target, which tells practitioners how much to trust any conformal tool beyond the cocktail sizes it was calibrated on.

As a final downstream illustration, we convert the stratified sets into a certify-or-refer rule in the spirit of Act or Defer, where a cocktail is certified as safe only if its set excludes the adverse label. We then report how many cocktails each method certifies at a fixed false-certification budget. This states the clinical cost of each method's coverage in a single number.