Post-Hoc Risk Control for Distribution-Free Prediction

This direction is merged into another direction: post-selection inference bias. Its last version stays readable and citable here.

Version 1 of 18 papers16 conceptsfrozen 2026-10-04

In traditional uncertainty quantification, a practitioner must decide on a "safety level" (often denoted as $\alpha$) before ever looking at their data. For example, in Conformal Prediction, you might pre-specify that you want 95% coverage. But in the real world, this is a blind choice. You don't know if 95% coverage will result in a small, useful prediction set or a massive, useless one until you see the calibration results.

The natural human instinct is to "peek": look at the trade-off between risk (error rate) and cost (efficiency), then pick the best-looking operating point. However, as Drew T. Nguyen, Anastasios N. Angelopoulos, and Stephen Bates demonstrate in Data-Adaptive Tradeoffs among Multiple Risks in Distribution-Free Prediction, this post-hoc selection introduces a "peeking effect." By choosing a threshold that performs well on your specific calibration set, you overfit the uncertainty bounds, invalidating the statistical guarantees and leading to higher-than-expected errors in production.

A new wave of research is solving this by developing frameworks that remain valid even when the user chooses their error tolerances after seeing the data.

The Problem of Selection Bias

The core issue is selection bias. When we use a calibration set to find a threshold, we assume that the performance on that set will generalize. If we pick the threshold $\alpha$ after seeing the calibration data, we are effectively searching for a lucky fluctuation in the data that makes our model look safer or more efficient than it actually is.

Etienne Gauthier, Francis Bach, and Michael Jordan note in Backward Conformal Prediction that this prevents practitioners from dynamically tuning the balance between reliability and set size. Similarly, Guneet S. Dhillon and colleagues argue in E-Scores for (In)Correctness Assessment of Generative Model Outputs that traditional p-value-based guarantees are fragile; if you modify your confidence threshold after seeing a Large Language Model's output, you risk underestimating the rate of hallucinations.

Strategy 1: Uniform Confidence Bounds

One way to allow post-hoc selection is to make the statistical guarantee "uniform." Instead of a bound that holds for a single pre-chosen $\alpha$, researchers construct bounds that hold simultaneously for all possible thresholds.

In Data-Adaptive Tradeoffs among Multiple Risks in Distribution-Free Prediction, Nguyen and Jordan develop uniform confidence bounds for monotone risk functions. Because the bound is valid across the entire range of possible risks, a user can explore the entire trade-off curve and pick any point without "breaking" the math. The guarantee is robust because it accounts for the fact that the user might have picked the most optimistic-looking threshold.

Strategy 2: Certified Pareto Frontiers via Data Splitting

Another approach, championed by Shixiang Zhu and Wenbin Zhou, involves a two-stage "Inverse Conformal Risk Control" process. In their framework, CREME, they explicitly separate the exploration of trade-offs from the certification of the final choice.

As detailed in Calibrating Decision Robustness via Inverse Conformal Risk Control, the process works like this:

  1. Partition the calibration data into two independent sets.
  2. Generate a Trade-off Curve using the first set. This allows the human decision-maker to see the "Pareto frontier"—how much efficiency (cost) they must sacrifice to gain more safety (robustness).
  3. Recalibrate and Certify the chosen point using the second, "untainted" data set.

By using this second set to provide the final performance certificate, Zhu and Zhou ensure that the selection bias from the first step does not leak into the final safety guarantee. This provides a "Certified Pareto Frontier" where every point on the curve has a mathematically rigorous backing.

Strategy 3: E-Values for Anytime-Valid Guarantees

A third direction replaces traditional p-values with e-values (and their counterparts, e-scores). Unlike p-values, which are tied to a fixed significance level, e-values are "anytime-valid."

In Adaptive Coverage Policies in Conformal Prediction, Elodie Gauthier and Michael I. Jordan use e-values to train neural networks that pick instance-specific coverage levels. Because e-values maintain their validity under data-dependent stopping and thresholding, they allow for "adaptive coverage policies" that can minimize prediction set sizes on the fly.

Similarly, Dhillon et al. use e-scores to bound the error of generative models. This allows a user to interactively adjust how "strict" they want the hallucination filter to be after seeing the model's scores, without falling into the trap of "p-hacking" their own uncertainty estimates.

Why It Works: Decoupling Selection from Estimation

The common thread across these papers is the decoupling of the selection of a policy from the estimation of its risk. Whether through uniform bounds that cover all policies, data-splitting that isolates the selection, or e-values that are inherently robust to thresholding, these methods allow for a "human-in-the-loop" or "data-driven" optimization process.

The result is a more flexible form of machine learning: one where we don't have to guess our risk tolerance in the dark, but can instead navigate the trade-offs between safety and efficiency with a guaranteed map.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_post_hoc_risk_control_for_distribution_f_ab0dd672,
  title = {Post-Hoc Risk Control for Distribution-Free Prediction},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/post-hoc-risk-control-for-distribution-free-prediction/rdr_ab0dd67232044ff292a4db4c43250c43},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_post_hoc_risk_control_for_distribution_f_ab0dd672_v1,
  title = {Post-Hoc Risk Control for Distribution-Free Prediction},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/post-hoc-risk-control-for-distribution-free-prediction/rdr_ab0dd67232044ff292a4db4c43250c43/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}