Distribution-Free Prediction Sets with Random Effects

Robin Dunn, Larry Wasserman, Aaditya Ramdas

Abstract

We consider the problem of constructing distribution-free prediction sets when there are random effects. For iid data, prediction sets can be constructed using the method of conformal prediction. The validity of this prediction set hinges on the assumption that the data are exchangeable, which is not true when there are random effects. We extend the conformal method so that it is valid with random effects. We develop a CDF pooling approach, a single subsampling approach, and a repeated subsampling approach to construct conformal prediction sets in unsupervised and supervised settings. We compare these approaches in terms of coverage and average set size. We recommend the repeated subsampling approach that constructs a conformal set by sampling one observation from each distribution multiple times. Simulations show that this approach has the best balance between coverage and average conformal set size.

Introduction

In statistical inference and machine learning, point estimates often fail to convey the inherent uncertainty of a prediction. Prediction sets—intervals or regions expected to contain a future observation with a specified probability (e.g., 90%)—offer a more robust alternative. Conformal prediction has emerged as a leading framework for generating these sets because it is "distribution-free," meaning it provides coverage guarantees without assuming a specific parametric form (like a Gaussian distribution) for the data.

However, the standard conformal framework relies on the assumption of exchangeability, which implies that the data points are independent and identically distributed (i.i.d.). In many real-world applications, data is hierarchical or grouped. For instance, multiple clinical measurements taken from the same patient are more similar to each other than to measurements from a different patient. This "random effect" violates exchangeability, rendering standard conformal methods invalid. This paper extends conformal prediction to handle such two-layer hierarchical models.

Methodology

The authors address two primary prediction tasks:

  1. Task 1 (New Subject): Predicting an outcome for a subject not present in the training data.
  2. Task 2 (Existing Subject): Predicting a new outcome for a subject already represented in the training set.

To address these, three main methods are proposed:

1. CDF Pooling

This method involves calculating the empirical Cumulative Distribution Function (CDF) for each group or subject. These individual CDFs are then averaged to create a pooled CDF, which determines the prediction boundaries. While computationally efficient and capable of producing narrow prediction sets, this method is only asymptotically valid—it requires a large number of groups to guarantee target coverage.

2. Single Subsampling

To restore exchangeability, one can randomly select exactly one observation from each of the $k$ groups. This resulting subsample consists of independent observations, allowing for the application of standard conformal prediction. While theoretically sound, this approach is statistically inefficient as it discards the majority of the available data, leading to high variance in the resulting intervals.

This approach builds on single subsampling by repeating the process $B$ times. For each subsample, a p-value is calculated; these p-values are then averaged to form a final test statistic. This "bagging-like" approach stabilizes the prediction sets and provides robust coverage guarantees even when the number of groups is small.

James-Stein Shrinkage for Task 2

For predictions involving existing subjects, the authors incorporate James-Stein shrinkage. By shrinking individual group means toward the global mean, the method "borrows strength" from the entire dataset. This reduces the size of the prediction sets significantly while maintaining the required coverage, particularly when within-group variance is high.

Experimental Evaluation

The methods were evaluated using both synthetic simulations and a real-world dataset.

Synthetic Simulations

The researchers tested the methods against various data distributions (Normal, Beta, and Exponential) and varying group sizes. These simulations were designed to test the limits of the coverage guarantees under both balanced and unbalanced data structures.

Sleep Deprivation Case Study

The practical utility of the methods was demonstrated using a sleep deprivation study involving 18 drivers. The dataset tracked reaction times over ten days of sleep restriction. This provided a supervised learning context where reaction times were predicted based on the number of days of sleep deprivation and baseline performance.

Key Results and Insights

The study yielded several critical findings regarding the trade-offs between different conformal strategies:

  • Reliability of Repeated Subsampling: In simulations, the repeated subsampling approach consistently met the target coverage (e.g., 90%) even with as few as $k=20$ groups. It proved to be the most reliable method for small-to-medium datasets.
  • Efficiency of CDF Pooling: CDF pooling consistently produced the smallest (most precise) prediction sets. However, it tended to "undercover" (provide less than the promised 90% coverage) when the number of groups was below 35.
  • The Power of Shrinkage: In Task 2, the application of James-Stein shrinkage allowed for much tighter prediction intervals without sacrificing the distribution-free guarantee.
  • Stability: In the sleep study, repeated subsampling achieved approximately 95% coverage for a 90% target. While slightly conservative, it was significantly more stable than single subsampling, which produced highly variable interval lengths depending on which specific data points were sampled.

Conclusion

This work extends the reach of conformal prediction to the common scenario of hierarchical data. By utilizing repeated subsampling, practitioners can generate valid, stable prediction sets even when the i.i.d. assumption is violated by random effects. While CDF pooling offers smaller sets in large-sample limits, repeated subsampling remains the recommended "gold standard" for its robust coverage guarantees across varying group sizes. Future research may further explore the asymptotic efficiency of these sets and extend supervised methods for existing subjects.

Summary updated

Abstract

We consider the problem of constructing distribution-free prediction sets when there are random effects. For iid data, prediction sets can be constructed using the method of conformal prediction. The validity of this prediction set hinges on the assumption that the data are exchangeable, which is not true when there are random effects. We extend the conformal method so that it is valid with random effects. We develop a CDF pooling approach, a single subsampling approach, and a repeated subsampling approach to construct conformal prediction sets in unsupervised and supervised settings. We compare these approaches in terms of coverage and average set size. We recommend the repeated subsampling approach that constructs a conformal set by sampling one observation from each distribution multiple times. Simulations show that this approach has the best balance between coverage and average conformal set size.
BibTeX
@article{dunn2018distribution,
  author = {Robin Dunn and Larry Wasserman and Aaditya Ramdas},
  title = {{Distribution-Free Prediction Sets with Random Effects}},
  journal = {Journal of the American Statistical Association},
  year = {2018},
}
Figure