When a brain-stimulation or other device trial reports a biomarker cutoff for "who will respond," that cutoff is usually chosen by sorting patients in the active arm into responders and non-responders. The cutoff is then the point on the biomarker that best separates the two groups, typically by maximizing sensitivity plus specificity (the Youden index). Two problems make such cutoffs untrustworthy. First, a convincing sham device, one that clicks and tingles like the real thing, produces real improvement of its own, so active-arm "responders" include many people who would have improved on sham. Second, patients keep taking antidepressants, anticonvulsants or sedatives during the trial, and these drugs shift the biomarker itself. A cutoff on raw heart-rate variability or connectivity values can end up separating medication regimens rather than device responsiveness. On top of both problems, these trials enroll 20 to 70 people. Empirical cutoffs at that size jump around and have optimistically biased accuracy.
We propose to estimate the cutoff from a different quantity: the active-minus-sham difference in benefit, expressed as a smooth function of a medication-adjusted biomarker score. Each patient's biomarker is first converted into their rank among patients on the same drug regimen. A semiparametric density-ratio model pools all regimens and both arms to keep that rank stable in small samples. The cutoff is then the rank at which the device's benefit over sham crosses a clinically meaningful margin. Its uncertainty comes from inference designed for small samples, not from the naive bootstrap. Put simply, we choose the cutoff by asking where the real device beats the sham device, after putting everyone's biomarker on a scale the drugs cannot shift.
Core Claim
Our central claim is that cutoffs chosen from the active arm alone pick up two kinds of signal that have nothing to do with the device. One is placebo responsiveness. The other is drug-induced distribution shift. A cutoff defined by a sham-contrasted benefit curve on a medication-adjusted scale removes both, and it can be estimated with calibrated confidence intervals at N ≈ 30–70 by borrowing strength across arms and regimens.
The method has two ingredients, and each is supported by evidence in the file. Sham arms are physiologically active: in Mori et al.'s upper-limb neuropathic pain trial, electrical scalp stimulation in the sham arm produced pain reduction of 0.58 ± 0.29 against 0.84 ± 0.31 for active treatment (p = 0.54). Medications shift the biomarkers in question. Antidepressants alter the autonomic coupling metrics in Zhao et al., and sedatives distort heart-rate variability baselines in Duong et al..
The closest existing work is the RMST adaptive enrichment design, which already chooses a cutoff by contrasting treatment against control. It assumes, however, that the biomarker reflects host biology alone, it uses a drug comparator rather than an active sham, and it targets large oncology trials. We keep its treatment-contrast logic and add two things it lacks: medication adjustment, and inference built for very small samples.
Research Program
Let $Y$ be clinical improvement, $A \in \{1, 0\}$ indicate active versus sham device, $X$ the baseline biomarker, and $M$ the medication regimen (drug class and dose). Instead of $X$ we use the placement value $U = F_{X \mid M}(X)$, a patient's rank within the biomarker distribution of patients on the same regimen. To estimate these regimen-specific distributions from few patients per regimen, we borrow the density ratio model of Liu, Tian, Wang and Li. That model links each group's density to a shared baseline through an exponential tilt. Liu et al. use it for two diagnostic groups; we extend it to many regimens, with the tilt depending on dose:
$$f_{m}(x) = \exp\{\alpha(m) + \beta(m)^\top q(x)\}\, f_0(x),$$
Here $f_0$ is a shared baseline density, $f_m$ is the biomarker density under regimen $m$, $q(x)$ is a low-dimensional basis such as $(x, x^2)$, and $\alpha(m), \beta(m)$ are linear in dose. Liu et al. report that this pooling shrinks a 95% joint confidence region from area 40.605 (fully nonparametric) to 3.889 at N = 100. That gain is exactly what our setting needs.
The target is the sham-contrasted benefit curve $\Delta(u) = E[Y \mid A=1, U=u] - E[Y \mid A=0, U=u]$. We estimate it with a penalized spline interaction model. The cutoff is
$$u^\ast = \inf\{u : \Delta(u) \ge \delta\},$$
where $\delta$ is a prespecified minimal clinically important difference over sham. The cutoff therefore answers a clinical question ("above what rank does the device beat sham by a meaningful amount?") rather than a classification question. It also avoids the step-function instability of the empirical Youden cutoff.
Inference proceeds in two layers. A smoothed bootstrap after Bertail, Clémençon and Vayatis resamples the full pipeline, including the density ratio fit. They showed that kernel smoothing raises ROC-curve band coverage from 90.3% to 93.1% and improves convergence from $n^{-1/4}$ to about $n^{-2/5}$. Separately, the interaction test that comes before any cutoff is reported carries a closed-form error rate.
The third component is a power analysis for active shams. We write the sham's physiological activity as a fraction $\rho$ of the active effect. We then derive and simulate the sample size needed to detect a nonzero slope of $\Delta(u)$ and to place $u^\ast$ within a given width, as a function of $\rho$ and of how much variance the medication shifts add. This turns the "active sham paradox" into a design quantity that trialists can budget for.
Baselines
Naive reference. The simplest reference point is a median split, or a fixed literature cutoff, on the raw biomarker in the active arm. It is evaluated on the same simulated and real cohorts. It is the cheapest thing a trialist would do, and any method that fails to beat it on out-of-sample benefit separation is not worth its complexity.
Current practice. The standard reported method is an empirical Youden cutoff on raw $X$ in the active arm, with the same responder definition. Xu et al. use this style of cutoff for 1,5-anhydro-D-glucitol, reporting sensitivity 56.20% and specificity 66.36%, as do Huang et al. for inflammatory ratios. We will strengthen this baseline as much as possible by giving it the smoothed bootstrap for its intervals, so that any gain we show is not merely a gain in inference.
Strongest existing competitor. This is the threshold-detection stage of the RMST adaptive enrichment design, adapted from survival to continuous outcomes and applied to raw $X$ with sham as the control arm. It is the strongest competitor because it already contrasts arms. Its reported result is in sample-size units: 845 versus 940 patients for 90% power on the JAVELIN Lung 200 re-design, with family-wise type I error below 2.5%. No reported result was found on Lacuna for any method estimating sham-contrasted cutoffs in device trials.
Ablations. Two ablations isolate our ingredients. The first keeps the sham contrast and the spline but uses raw $X$, which isolates medication adjustment. The second uses medication-adjusted $U$ with a Youden cutoff in the active arm only, which isolates the sham contrast.
Fair comparison. All methods see the same patients, the same responder or outcome definition, and the same nested cross-validation budget for any tuning. Any cutoff is evaluated on held-out folds. We count our method as better than the strongest competitor if it achieves both of the following. First, its held-out cutoff must yield a larger observed active-minus-sham benefit among biomarker-positive patients. Second, its cutoff confidence intervals must cover the true value at close to the nominal 95% in simulation where the baselines fall short, and must be no wider when coverage is matched.
Why not just…?
Why not just regress the biomarker on medication and use the residual? This is the most serious alternative for the medication part, and it may tie with our approach when there are few regimens and the drug effects are linear in the mean. It breaks down when drugs change the spread or shape of the biomarker distribution and not only its mean. That is plausible for sedative depression of autonomic tone (Duong et al.), and the density ratio model's quadratic basis captures it. Residualization also does nothing about placebo response. We include it as an extra baseline under simulated shift that affects the mean only and shift that affects shape. If it matches our method under realistic shape shifts, we will recommend it, since it is simpler.
Why not just test a continuous treatment-by-biomarker interaction and skip cutoffs? This is the right test for whether a biomarker predicts device benefit, and dichotomizing loses power. We therefore adopt it. Our cutoff is read off the same continuous interaction curve, and a cutoff is reported only if the interaction test passes. The cutoff is still needed because enrichment and clinical allocation are yes/no decisions. The added contribution is a medication-invariant scale and an honest interval on where the decision boundary lies.
Why not just enforce medication washout or use waitlist controls? Withholding medication is often impossible. Stimulation in treatment-resistant schizophrenia is an add-on to antipsychotics (Wei et al.), and sedation in tetanus keeps patients alive. Waitlist and treatment-as-usual controls underperform sham in that same meta-analysis, which means they leave placebo response inside the "device effect." Sham surgery produced lasting improvement in the AAV2-GAD Parkinson's trial. Removing the sham would hide the very effect we need to subtract.
Why not augment the small sample with synthetic patients? DE-CGAN reports 0.9360 accuracy against 0.9227 on real data alone for rTMS response prediction. However, its generator is conditioned only on response labels. It would therefore reproduce drug-linked and placebo-linked structure, and it provides no calibrated interval on a cutoff.
Decisive Experiments
Simulation design. The first study is a simulation calibrated to effect sizes reported in the file. Active-versus-sham contrasts span SMD 0.19 to 0.47, following Wei et al.. Sham activity ratios include $\rho \approx 0.7$, implied by Mori et al.'s 0.58/0.84. Sample sizes are 30, 69 and 150, matching the neuropathic pain and Alzheimer's rTMS cohorts. Medication shifts affect the location, scale or both of $X$, and are either independent of device benefit or correlated with placebo response.
Two decisive checks. First, when medication shifts $X$ but not $\Delta$, the active-arm Youden cutoff should drift with the mix of regimens across simulated sites, while $u^\ast$ should not. Second, when placebo response depends on $X$ but device-specific benefit does not, the active-arm methods should report a "predictive" cutoff, while our interaction test should hold its nominal error rate. Across 2,000 replications per cell we report cutoff bias, interval coverage and width, held-out benefit among biomarker-positive patients, and interaction-test power relative to the plain continuous interaction test. We will consider our method to have failed if it loses more than a small, prespecified margin of that test's power.
Power curves. We will produce sample-size curves as a function of $\rho$, and check them against the enrichment design's simulation setup (JAVELIN re-design). The point is to confirm that, when there is no sham activity and no medication effect, our method reduces to their behavior.
Real-data reanalysis. We will reanalyze patient-level data from sham-controlled rTMS cohorts if the authors share them. The natural candidates are the Osaka pain trial, the parietal Alzheimer's trial, and the NCG-TMS 2.0 heart–brain coupling data, which contain sham conditions and continuous ECG. The NCG-TMS participants are unmedicated, so that cohort serves as a negative control: the medication adjustment should change nothing there. In the medicated cohorts the prediction is falsifiable. Cutoffs on raw values should differ between regimen subgroups, and placement-value cutoffs should agree within their intervals. If patient-level data cannot be obtained, the simulation study stands as the primary evidence and we will say so plainly.
References
- Repetitive transcranial magnetic stimulation focusing on patients with neuropathic pain in the upper limb: a randomized sham-controlled parallel trial
- Cardiorespiratory Coupling Analysis Based on Entropy and Cross-Entropy in Distinguishing Different Depression Stages
- Heart Rate Variability as an Indicator of Autonomic Nervous System Disturbance in Tetanus
- Biomarker-guided adaptive enrichment design with threshold detection for clinical trials with time-to-event outcome
- Semiparametric Joint Inference for Sensitivity and Specificity at the Youden-Optimal Cut-Off
- On Bootstrapping the ROC Curve
- Plasma 1,5-anhydro-d-glucitol is associated with peripheral nerve function and diabetic peripheral neuropathy in patients with type 2 diabetes and mild-to-moderate hyperglycemia
- A new model of preoperative systemic inflammatory markers predicting overall survival of osteosarcoma: a multicenter retrospective study
- Non-invasive brain stimulation augmentation therapy for treatment-resistant schizophrenia: a systematic review and network meta-analysis
- Long-term follow-up of a randomized AAV2-GAD gene therapy trial for Parkinson’s disease
- DE-CGAN: Boosting rTMS Treatment Prediction with Diversity Enhancing Conditional Generative Adversarial Networks
- Precision Repetitive Transcranial Magnetic Stimulation Over the Left Parietal Cortex Improves Memory in Alzheimer’s Disease: A Randomized, Double-Blind, Sham-Controlled Study
- Target-Specificity and repeatability in neuro-cardiac-guided TMS for heart-brain coupling