Most published comparisons, whether a drug against placebo or one language model against another, end in a binary verdict: $p < 0.05$, therefore "significant." That verdict answers the wrong question. With enough data, a negligible difference becomes highly significant. A non-significant result is routinely read as proof of no effect. A $p$ of 0.08 gets reported as a "trend." The question a clinician or engineer actually needs answered has three possible answers. The effect may be larger than the smallest difference that matters, called the minimal clinically important difference (MCID). It may be safely inside that margin, so the two options are practically equivalent. Or the data may not yet say. This note proposes building confidence intervals designed to reach a confident answer to that three-way question as often as possible. They use prior knowledge, such as historical trials or results on related benchmarks, to sharpen the call. They still guarantee, for every possible true effect and whether or not the prior is right, that the verdict is wrong at most a fraction $\alpha$ of the time.
The technical opening comes from a family of methods called Frequentist, Assisted by Bayes (FAB). These methods use a prior distribution to shape a confidence interval while keeping exact repeated-sampling coverage. Existing FAB methods optimize one target: making the interval short on average around wherever the prior expects the effect to be. A short interval is not the same as a decisive one. An interval of width 1 centered exactly on the MCID boundary decides nothing. A wider interval sitting cleanly inside the equivalence band decides everything. Our claim is that optimizing directly for decisiveness against an MCID band is a different mathematical problem from optimizing for width, and that it yields measurably more correct three-way calls at identical worst-case error.
Core Claim
Let $\theta$ be the true effect, $\epsilon > 0$ the MCID, and $C(X)$ a confidence interval computed from data $X$. The interval must satisfy $P_\theta(\theta \in C(X)) \ge 1-\alpha$ for every $\theta$. Call the outcome decisive if $C(X)$ lies entirely inside $(-\epsilon, \epsilon)$ (practical equivalence), entirely above $\epsilon$ (meaningful benefit), or entirely below $-\epsilon$ (meaningful harm). Any decisive verdict that is wrong requires $\theta \notin C(X)$. So the three-way decision rule inherits error at most $\alpha$ under every $\theta$, regardless of the prior $\pi$. We propose choosing the interval to solve
$$\max_{C}\ \int P_\theta\big(C(X)\ \text{is decisive}\big)\,\pi(\theta)\,d\theta \quad \text{subject to}\quad P_\theta(\theta\in C(X))\ge 1-\alpha\ \ \forall\theta .$$
The contrast with FAB is structural. FAB minimizes prior-expected length. By Fubini's theorem, that objective splits into independent subproblems, one per candidate value $\theta'$: for each one, pick the acceptance region of the level-$\alpha$ test at $\theta'$ that is least probable under the prior's predicted data distribution. Our objective does not split this way. Claiming benefit requires rejecting every $\theta' \le \epsilon$ at once, which is a property of the union of acceptance regions rather than of any single one. The optimal design therefore trades coverage slack across $\theta'$ values jointly. Intuitively, it spends power at and near the boundaries $\pm\epsilon$, where verdicts are won or lost, rather than near the prior's center.
For the scalar effects that MCID problems usually involve, this is tractable. Parameterize each acceptance region by its lower and upper offsets $a(\theta'), b(\theta')$, tied together by the coverage constraint, and optimize these over a one-dimensional grid. This sidesteps the high-dimensional grid cost that limits FAB.
Research Program
The first component is construction and theory in the Gaussian known-variance model. That model is the setting where Bayes-assisted Confidence Regions: Focal Point Estimator and Bounded-influence Priors proves that FAB regions stay bounded when the prior has heavy (power-law) tails, and it is the template for treatment-difference and accuracy-difference estimates. We will compute the decision-optimal offset functions numerically. We will characterize when the resulting regions are intervals; if they are not, we use the convex hull, which preserves coverage. We will then prove three properties. The first is the guaranteed three-way error bound. The second is a degeneracy result: as the prior flattens, the procedure should reduce to the familiar $(1-2\alpha)$ interval-inclusion rule underlying the two one-sided tests (TOST) equivalence procedure. The third is a robustness result analogous to Cortinovis and Caron's: with heavy-tailed priors, decisiveness under severe prior–data conflict should fall back toward the no-prior procedure rather than collapse. The third result is a hypothesis we expect to inherit from their bounded-influence argument, not a given.
The second component is prior design for decisions. Existing FAB work, including FAB-PPI, uses horseshoe priors, which pile mass on a single point at zero. For an equivalence band, every value in $(-\epsilon, \epsilon)$ is equally "null," and a point spike can mislead near $\pm\epsilon$. We will compare spike-at-zero, band-uniform-with-heavy-tails, and empirical-Bayes priors fitted on held-out related problems. We will measure which ones raise decisiveness without inflating the worst-case loss.
The third component is a sequential extension, kept deliberately secondary. The prior-to-posterior ratio evaluated at a candidate value equals the marginal likelihood divided by the likelihood at that value. It is a nonnegative martingale, so Ville's inequality makes it valid under continuous monitoring, as exploited in PAC Mode Estimation using PPR Martingale Confidence Sequences and Confidence sequences for sampling without replacement. The verdict at time $n$ depends only on this ratio at the boundaries $\pm\epsilon$. This gives an anytime-valid stopping rule for "meaningful" or "equivalent." The prior affects only the stopping time, which we will try to minimize, as above, for the decision rather than for width. Anytime-valid, Bayes-assisted, Prediction-Powered Inference builds such sequences for scalar means but never against a margin. We reuse its machinery and change the target.
The $p$-value misinterpretation literature plays the role of motivation and audit, not mechanism. Not All Claims are Created Equal found that 86% of surveyed NLP researchers felt confident interpreting $p$-values while few could define them, and that no ACL-2018 papers used Bayesian assessment. Addressing Common Misuses and Pitfalls of P values documents the large-sample and "proof of the null" fallacies. The three-way output turns each fallacy into an explicit category. "Significant but trivial" becomes equivalent. A "trend" or "no effect" reading becomes ambiguous. Unlike a $p$-value, every category has a stated error rate. In the flat-prior limit, we will also extend the one-sided $p$-value/posterior identity of Reconnecting p-Value and Posterior Probability to the boundary tests at $\pm\epsilon$. That identity is shown for directional and point nulls; we aim to give equivalence verdicts a correct posterior reading. This is a modest corollary rather than the core contribution.
Baselines
The naive reference point is the reporting practice the literature criticizes: a point-null test at $\alpha=0.05$, read as "meaningful" when significant and "no difference" when not. We will run it on the same data only to count how many of its claims our method relabels as equivalent or ambiguous. It is not a valid three-way procedure, so it cannot "win" on error.
The prior-free frequentist baseline is TOST-style interval inclusion with the standard $(1-2\alpha)$ interval. It also includes the uniformly most powerful equivalence test for the Gaussian location problem, which is more powerful than TOST. Both use exactly the same data and $\alpha$. They require no tuning. Beating them means higher decisiveness when the prior is roughly right. Matching them within a pre-registered tolerance (say, two percentage points of decisiveness) means doing so when the prior is badly wrong.
The strongest existing Bayes-assisted competitor is a FAB confidence region from Bayes-assisted Confidence Regions, plugged into the same interval-inclusion rule, with the same prior family and hyperparameters. This is the baseline that isolates our key ingredient, since the only difference is the objective: decisiveness versus length. FAB-PPI reports that it "maintained the required 90% coverage rate" across its real-data suites. No reported result was found on Lacuna for the decisive-call rate of FAB intervals against any margin, so we will generate that number ourselves. Beating FAB means strictly higher prior-weighted decisiveness at equal guaranteed error, with a confidence interval on the gain across simulation replicates that excludes zero.
For the model-comparison setting, the frequentist margin baseline is the indifference-margin directional testing of Quantifying Ranking Uncertainty in LLM Benchmarks. That method uses $\delta=2\%$ and Holm correction, and it reports rank intervals spanning 3 to 5 positions for many models after filtering 27 non-informative subjects. We will run it on the same PromptEval subjects, model pairs, and margin. Beating it means more pairs resolved decisively, and tighter rank intervals as a consequence, under the same family-wise error target.
The pure Bayesian baseline is HyBayes, the highest-density-interval-versus-ROPE (region of practical equivalence) analysis from Not All Claims are Created Equal, along with the paired model of Bayesian Performance Comparison of Text Classifiers. It uses the same prior. Here we expect HyBayes to be more decisive, and the comparison concerns its worst-case error under misspecified priors.
Why not just…?
Why not just use TOST, or the most powerful equivalence test? These are valid and, without a prior, essentially optimal. Our method reduces to them when the prior is flat. The only claim is that real prior information can be converted into extra decisive calls without giving up the error guarantee. They cannot use priors at all, which matters most in small studies such as the ChAMP trial analyzed in Test of Significance for High-Dimensional Thresholds, with $n=138$. The simulation grid will show directly whether the gain is large enough to matter. If decisiveness gains under correct priors are tiny, the proposal fails on its own terms.
Why not just plug an existing FAB interval into the inclusion rule? This is the real threat, and we take it seriously. FAB already produces valid prior-shaped intervals, and inclusion-based decisions on them are valid. Our argument is that length minimization shrinks the interval near the prior center, while decisions are made at $\pm\epsilon$. When the prior is centered at zero with $\epsilon$ in its shoulder, the two objectives should diverge most. When $\epsilon$ is far from the prior's mass, they may nearly coincide. The head-to-head ablation varies $\epsilon$ relative to prior scale. If FAB-plus-inclusion matches us everywhere, the right conclusion is that FAB already solves the decision problem, and we would report that as the finding.
Why not just use the Bayesian ROPE analysis directly? It answers the three-way question in posterior terms and is more decisive by design. But it has no repeated-sampling guarantee. Bayes-assisted Confidence Regions shows that badly matched Gaussian priors make Bayesian regions behave pathologically under prior–data conflict. Regulatory and benchmark settings require error control. Our misspecification sweep will report HyBayes's worst-case three-way error rate. If that rate stays near $\alpha$ across realistic misspecification, the case for our method narrows to formal guarantees alone, and we will say so.
Why not just lower $\alpha$ or report effect sizes? Lower thresholds do not stop trivial effects from becoming significant at large $n$, and they do not create a way to affirm equivalence. A more principled use of the p-value? Not so fast argues that such recalibrations do not fix the underlying inferential error. We include $\alpha=0.005$ point-null testing in the audit to show how many trivially small differences it still flags as significant.
Proposed Experiments
The decisive simulation uses the Gaussian known-variance model. It crosses true effects (inside, near, and beyond $\pm\epsilon$), prior centers and scales (correct, shifted by one, three, and ten standard errors), $\epsilon$-to-standard-error ratios, and sample sizes. The primary metrics are worst-case three-way error over a fine grid of $\theta$, which must stay at or below $\alpha$; prior-weighted decisiveness; and decisiveness under each misspecification level. The first falsification check is any $\theta$ at which the error exceeds $\alpha$ beyond Monte Carlo noise. The second is failure to beat FAB-plus-inclusion anywhere in the grid. The third is a decisiveness loss relative to the uniformly most powerful test that does not shrink back toward zero as conflict grows, which would mean our bounded-influence conjecture fails.
On MMLU PromptEval (15 models, 57 subjects, 100 questions per subject, $\delta=2\%$), we build each subject's prior over pairwise accuracy differences from the other 56 subjects, a leave-one-subject-out empirical Bayes prior. We then compare resolved pairs and rank-interval widths against the Neuhof et al. procedure and FAB-plus-inclusion. Because the true differences are unknown, we validate error control by subsampling questions from subjects with large question pools and treating the full-pool difference as ground truth.
We also reanalyze two published comparisons. The first is ARC, BERT versus Reading Strategies, where the point-null test gave $p=0.0037$ and HyBayes reported a non-trivial probability that the difference was under 1%. The second is SVM-L1 versus SVM-L2 on 20 Newsgroups, reported as practically equivalent by ROPE. Both serve as qualitative checks that the guaranteed procedure reaches, or honestly declines to reach, the same verdicts. If the ChAMP data are accessible, we will apply the method to WOMAC pain change against published MCID values, with priors drawn from the trial's baseline literature.
The sequential arm replays the PromptEval questions in random order. It compares the stopping time to a decisive verdict for the prior-to-posterior-ratio rule with a decision-targeted prior, a width-targeted prior, and no prior. Cumulative error must stay below $\alpha$ across 1,000 replays.
References
- Bayes-assisted Confidence Regions: Focal Point Estimator and Bounded-influence Priors
- FAB-PPI: Frequentist, Assisted by Bayes, Prediction-Powered Inference
- PAC Mode Estimation using PPR Martingale Confidence Sequences
- Confidence sequences for sampling without replacement
- Anytime-valid, Bayes-assisted, Prediction-Powered Inference
- Not All Claims are Created Equal: Choosing the Right Statistical Approach to Assess Hypotheses
- Addressing Common Misuses and Pitfalls of P values in Biomedical Research
- Reconnecting p -Value and Posterior Probability Under One- and Two-Sided Tests
- Quantifying Ranking Uncertainty in LLM Benchmarks
- Bayesian Performance Comparison of Text Classifiers
- Test of Significance for High-Dimensional Thresholds with Application to Individualized Minimal Clinically Important Difference
- A more principled use of the p -value? Not so fast: a critique of Colquhoun’s argument