Abstract
The ipd package performs valid statistical inference on predicted data (IPD) using recent methods, where for a subset of the data, the outcomes have been predicted by an algorithm. It provides a wrapper function with specified defaults for the type of model and method to be used for estimation and inference. Further, it provides methods for tidying and summarizing results. (Salerno et al., 2024).
Introduction: The Labeling Bottleneck
In modern data science, researchers often face a "labeling bottleneck." While massive datasets of millions of observations may be available, only a tiny fraction typically contains "gold-standard" ground truth labels due to the high cost or difficulty of acquisition. A common workaround is to train a machine learning model on the small labeled set and use it to predict outcomes for the remainder—the "unlabeled" set.
However, treating these "silver-standard" predictions as ground truth in downstream statistical analysis is statistically dangerous. Standard tests assume data is observed, not estimated. Using raw predictions typically leads to biased estimates and "overconfident" results, characterized by deceptively narrow confidence intervals and small p-values. The ipd package addresses this by providing tools to "rectify" these inferences, combining the precision of large-scale predictions with the mathematical validity of gold-standard labels.
Unified Framework for Rectification
The core contribution of ipd is its role as a high-level wrapper for several recently developed statistical frameworks. Instead of requiring users to implement complex mathematics from scratch, the package provides a consistent interface via the ipd() function.
Key methodologies included are:
- Post-Prediction Inference (PostPI): Employs analytic and bootstrap corrections to adjust for the relationship between observed and predicted values.
- Prediction-Powered Inference (PPI & PPI++): Uses a "rectifier" approach to ensure that even if the underlying ML model is biased or a "black-box," the final inference remains valid.
- Assumption-Lean and Data-Adaptive Post-Prediction Inference (PSPA): Adaptively weights data to minimize variance.
The package supports a wide range of downstream models, including mean estimation, quantiles, Ordinary Least Squares (OLS), Logistic regression, and Poisson regression. It also integrates with the "tidy" ecosystem, offering tidy(), glance(), and augment() methods.
Key Insights: The Rectifier and Power Tuning
The package treats ML predictions not as the final word, but as a "noisy" version of the truth. By comparing predictions to ground truth on a small subset, it calculates a "rectifier" that shifts and scales the final estimates to ensure they are unbiased.
Advanced methods like PPI++ include a power-tuning parameter ($\hat{\lambda}$) that determines how much to trust the predictions versus the labeled data. The package includes specialized functions to optimize this balance automatically.
Benchmarking and Distributional Inference
To facilitate research, the package includes the simdat function, a data generation tool designed for benchmarking IPD methods. It allows users to simulate training, labeled, and unlabeled sets to test how different methods perform under varying levels of model bias or signal-to-noise ratios. Beyond point estimates, the package can compute "rectified" Empirical Cumulative Distribution Functions (CDFs), allowing for a granular look at how predictions differ from reality across the entire data range.
Conclusion
The ipd package provides the infrastructure to achieve "validity under any model." Even if a prediction algorithm is poor, these methods produce confidence intervals that cover the true parameter at the advertised rate. When the prediction model is accurate, the package yields significant "precision gain," effectively turning unlabeled data into a valuable statistical resource rather than a source of bias.
Summary updated