TL;DR
This paper introduces "postprediction inference" (postpi), a novel method to correct statistical inference when using machine learning-predicted outcomes as dependent variables. It addresses the common issue of biased estimates and deflated standard errors that arise when predicted outcomes are treated as observed, offering a way to achieve more reliable results across various machine learning models.
What Problem Does This Paper Address?
In many scientific fields, particularly medicine and public health, machine learning (ML) models are increasingly used to predict outcomes based on observable data. For example, ML can predict a patient's likelihood of developing a disease or the cause of death based on reported symptoms. However, a critical challenge arises when these predicted outcomes are then used in subsequent statistical analyses (like regression models) as if they were observed outcomes. This practice, termed "postprediction inference," can lead to significant statistical problems: the estimated coefficients might be biased, the standard errors will likely be too small, and consequently, hypothesis tests and p-values become unreliable, leading to an increased risk of false positives. This paper aims to bridge this gap by providing a principled way to correct for the uncertainty introduced by using predicted outcomes.
Methods
The core of the proposed "postprediction inference" (postpi) method lies in leveraging a standard ML data splitting strategy: training, testing, and validation sets.
- Prediction Model Training: An ML model (e.g., random forest, deep neural network) is trained on the training set to predict the outcome ($y$) from covariates ($x$). This model, $f(x)$, is then used to generate predicted outcomes ($y_p$).
- Relationship Estimation: Crucially, the authors observe that the relationship between the observed outcome ($y$) and the predicted outcome ($y_p$) can often be modeled by a simple, low-dimensional function (e.g., linear regression). This relationship is estimated using the testing set, where both $y$ and $y_p$ are available. This estimated relationship, $r(y_p, y)$, acts as a correction factor.
- Corrected Inference: In the validation set (or any future dataset where only covariates are observed), the ML model generates predicted outcomes ($y_p$). The estimated relationship from the testing set is then used to adjust the inference (e.g., regression coefficients, standard errors) performed on these predicted outcomes.
The paper offers two main approaches for this correction:
- Bootstrap-Based Correction: A flexible method that uses bootstrapping to simulate outcomes that account for both the prediction error and the sampling variability. This approach is applicable to a wide range of outcome types (continuous, categorical) and inferential models.
- Analytical Correction: A more efficient method that provides closed-form solutions for bias and variance correction. This approach is more restrictive, assuming continuous outcomes, a linear relationship between observed and predicted outcomes, and a linear inferential model.
Datasets
The paper utilizes several datasets for evaluation:
- Simulated Data: Both continuous and binary outcome simulations were generated to demonstrate the core issues of postprediction inference and the effectiveness of the proposed methods.
- Recount2 Project Data: This dataset, containing RNA-seq gene expression data, was used to predict tissue types (breast vs. adipose). This application highlights the method's utility in genomics.
- Population Health Metrics Research Consortium (PHMRC) Data: This dataset was used to predict causes of death based on reported symptoms. This application showcases the method's relevance in public health and epidemiology.
The paper does not introduce new benchmark datasets but rather applies its methods to existing, well-characterized datasets to demonstrate their broad applicability.
Key Insights and Ideas
- The "Low-Dimensional Representation" Hypothesis: The most significant insight is the observation that even complex ML models often produce predictions that have a surprisingly simple, often linear, relationship with the true outcome. This makes correction feasible without needing to understand the internal workings of every ML algorithm.
- Decoupling Prediction and Inference: The method elegantly separates the task of prediction (using complex ML) from the task of inference (using simpler statistical models), allowing each to be optimized independently while still accounting for their interaction.
- Leveraging the Testing Set: The testing set, often used for hyperparameter tuning or model selection, is repurposed here to estimate the crucial relationship between observed and predicted outcomes, providing the key to correction.
- Generalizability: The postpi approach is designed to be model-agnostic, meaning it can be applied to predictions from any ML algorithm, from simple linear models to complex deep neural networks.
Key Results
The paper demonstrates that the postpi approach significantly improves statistical inference compared to the "no correction" baseline:
- Bias Reduction: Postpi methods consistently reduce bias in coefficient estimates across simulated and real-world applications. For instance, in the binary simulation, the RMSE of estimates was reduced from 2.94 to 0.53.
- Improved Standard Error Estimation: Standard errors are corrected to be more accurate, moving away from the overly small values produced by uncorrected inference. In the continuous simulation, the RMSE for SEs dropped from 0.088 to 0.015.
- More Reliable Hypothesis Testing: Corrected t-statistics and p-values lead to better control of Type I error rates. In the binary simulation, the RMSE for t-statistics was reduced from 2.06 to 2.04 (with a slight conservative bias, which is acceptable).
- Practical Applications: The methods showed substantial improvements in inferring associations between gene expression and tissue types, and in describing cause-of-death distributions. For example, in the tissue type prediction, the RMSE of estimates was reduced from 0.36 to 0.08.
Limitations and Future Work
- Assumption of Data Distribution: Both methods assume that the training, testing, and validation sets originate from the same data-generating distribution. If this assumption is violated (e.g., due to dataset shift), the corrections may not hold.
- Fitness of the Relationship Model: The effectiveness of postpi relies on the ability of the chosen relationship model (e.g., linear regression) to accurately capture the link between observed and predicted outcomes. If this relationship is very weak or complex, the correction might be less effective.
- Covariate Subset: The current methods assume that the covariates used in the downstream inferential model are a subset of those used in the prediction model. Introducing new covariates not seen during prediction training could pose challenges.
- Future Work: The authors suggest exploring methods to handle violations of the data distribution assumption, such as using data normalization techniques. They also highlight the potential for extending the approach to more complex inferential models beyond generalized linear models. The availability of an open-source R package facilitates wider adoption and further development.
Summary updated