TL;DR
This paper introduces "postpi," a novel framework for correcting statistical inference when using machine learning-predicted outcomes instead of actual observed outcomes. It addresses the common but problematic practice of treating predictions as ground truth, which leads to biased results, and offers a flexible, generalizable solution applicable across various machine learning models and data types.
What Problem Does This Paper Address?
In many scientific fields, particularly medicine and public health, machine learning models are increasingly used to predict outcomes (e.g., disease risk, cause of death) based on observable data. However, a common practice is to then use these predicted outcomes directly in subsequent statistical analyses (like regression models) as if they were the observed outcomes. This "post-prediction inference" is problematic because it ignores the inherent uncertainty and potential biases introduced by the prediction model. This can lead to:
- Biased estimates: The estimated relationships between variables might be systematically wrong.
- Deflated standard errors: Confidence intervals and p-values become overly optimistic, suggesting more certainty than exists.
- Inflated false positive rates: Researchers are more likely to incorrectly conclude that a relationship exists when it doesn't.
The paper aims to bridge this gap by providing a statistically sound method to correct for these issues, ensuring more reliable inference when working with predicted outcomes.
Methods
The core of the "postpi" approach is to leverage the relationship between the observed and predicted outcomes, which is estimated on a separate "testing" dataset. The authors propose a framework that naturally fits into the standard machine learning data splitting paradigm:
- Training Set: Used to train the initial prediction model (e.g., a random forest, neural network, etc.) to predict the outcome ($y_p$) from covariates ($x$).
- Testing Set: Used to estimate the relationship between the observed outcomes ($y$) and the predicted outcomes ($y_p$) from the training set. This relationship is modeled using a flexible function, $y = k(y_p)$.
- Validation Set: Used for the downstream inferential analysis (e.g., fitting a regression model). Here, only covariates ($x$) are available, and predicted outcomes ($y_p$) are used. The relationship learned from the testing set is then applied to correct the inference on this validation set.
The paper offers two main correction methods:
- Bootstrap Correction: A flexible approach that can handle various outcome types (continuous, categorical) and inferential models. It involves resampling from the estimated relationship model to simulate corrected outcomes and then performing the inference.
- Analytical Derivation: A more efficient method that provides a closed-form solution but requires stronger assumptions, specifically that the outcome is continuous and approximately normally distributed, and the relationship model is linear.
Datasets
The paper uses simulated data for both continuous and binary outcomes to demonstrate the methods. Additionally, it applies the "postpi" approach to two real-world datasets:
- Recount2 Project: Gene expression data used to predict tissue types (breast vs. adipose).
- Population Health Metrics Research Consortium (PHMRC): Verbal autopsy data used to predict causes of death.
These datasets represent distinct scientific domains and highlight the broad applicability of the proposed methods.
Key Insights and Ideas
- The "post-prediction inference" problem is widespread: The paper highlights that using predicted outcomes as if they were observed is a common practice across many scientific disciplines, leading to potentially flawed conclusions.
- A simple relationship often exists: Crucially, the authors observe that even complex machine learning models often exhibit a relatively simple, low-dimensional relationship between the predicted and observed outcomes. This relationship can be effectively modeled and leveraged for correction.
- Decoupling prediction from inference: The "postpi" framework separates the task of prediction (maximizing accuracy) from the task of inference (correctly quantifying uncertainty), which is a key conceptual shift.
- Generalizability: The methods are designed to be model-agnostic, meaning they can be applied regardless of the specific machine learning algorithm used for prediction.
Key Results
- Bias and Variance Correction: Both the analytical and bootstrap methods successfully correct for bias in parameter estimates and improve the accuracy of standard error estimation compared to the uncorrected approach.
- Improved Hypothesis Testing: Corrected p-values and t-statistics are shown to be more reliable, leading to better control of Type I error rates.
- Performance across models: The methods demonstrate effectiveness across various prediction models, including k-nearest neighbors, random forests, SVMs, and neural networks.
- Real-world application success: The "postpi" approach showed significant improvements in accuracy and reliability when applied to gene expression data for tissue type prediction and verbal autopsy data for cause of death prediction. For instance, in the continuous simulation, RMSE for standard errors was reduced from 0.088 (no correction) to 0.015 (analytical derivation and parametric bootstrap).
Limitations and Future Work
- Dependence on relationship model fitness: The accuracy of the correction relies on how well the estimated relationship between observed and predicted outcomes captures the true relationship. If predictions are from "weak learners" with low correlation to observed outcomes, the correction may be less effective.
- Assumption of same data generating distribution: The methods assume that the training, testing, and validation sets share the same underlying data generating process. If this assumption is violated, inference on bootstrapped values might not be reliable. The authors suggest data normalization techniques as a potential solution.
- Future Work: The authors suggest exploring data normalization to address violations of the data generating distribution assumption. They also highlight the release of an open-source R package ("postpi") to make their methods accessible to the broader research community.
Summary updated