Methods for correcting inference based on outcomes predicted by machine learning

Siruo Wang, Tyler H. McCormick, Jeffrey T. LeekProceedings of the National Academy of Sciences 2020 · DOI

TL;DR

This paper introduces "postprediction inference" (postpi), a novel method to correct statistical inference when using machine learning-predicted outcomes as dependent variables. It addresses the common issue of biased estimates and deflated standard errors that arise when predicted outcomes are treated as observed, offering a way to achieve more reliable results across various machine learning models.

What Problem Does This Paper Address?

In many scientific fields, particularly medicine and public health, machine learning (ML) models are increasingly used to predict outcomes based on observable data. For example, ML can predict a patient's likelihood of developing a disease or the cause of death based on reported symptoms. However, a critical challenge arises when these predicted outcomes are then used in subsequent statistical analyses (like regression models) as if they were observed outcomes. This practice, termed "postprediction inference," can lead to significant statistical problems: the estimated coefficients might be biased, the standard errors will likely be too small, and consequently, hypothesis tests and p-values become unreliable, leading to an increased risk of false positives. This paper aims to bridge this gap by providing a principled way to correct for the uncertainty introduced by using predicted outcomes.

Methods

The core of the proposed "postprediction inference" (postpi) method lies in leveraging a standard ML data splitting strategy: training, testing, and validation sets.

  1. Prediction Model Training: An ML model (e.g., random forest, deep neural network) is trained on the training set to predict the outcome ($y$) from covariates ($x$). This model, $f(x)$, is then used to generate predicted outcomes ($y_p$).
  2. Relationship Estimation: Crucially, the authors observe that the relationship between the observed outcome ($y$) and the predicted outcome ($y_p$) can often be modeled by a simple, low-dimensional function (e.g., linear regression). This relationship is estimated using the testing set, where both $y$ and $y_p$ are available. This estimated relationship, $r(y_p, y)$, acts as a correction factor.
  3. Corrected Inference: In the validation set (or any future dataset where only covariates are observed), the ML model generates predicted outcomes ($y_p$). The estimated relationship from the testing set is then used to adjust the inference (e.g., regression coefficients, standard errors) performed on these predicted outcomes.

The paper offers two main approaches for this correction:

  • Bootstrap-Based Correction: A flexible method that uses bootstrapping to simulate outcomes that account for both the prediction error and the sampling variability. This approach is applicable to a wide range of outcome types (continuous, categorical) and inferential models.
  • Analytical Correction: A more efficient method that provides closed-form solutions for bias and variance correction. This approach is more restrictive, assuming continuous outcomes, a linear relationship between observed and predicted outcomes, and a linear inferential model.

Datasets

The paper utilizes several datasets for evaluation:

  • Simulated Data: Both continuous and binary outcome simulations were generated to demonstrate the core issues of postprediction inference and the effectiveness of the proposed methods.
  • Recount2 Project Data: This dataset, containing RNA-seq gene expression data, was used to predict tissue types (breast vs. adipose). This application highlights the method's utility in genomics.
  • Population Health Metrics Research Consortium (PHMRC) Data: This dataset was used to predict causes of death based on reported symptoms. This application showcases the method's relevance in public health and epidemiology.

The paper does not introduce new benchmark datasets but rather applies its methods to existing, well-characterized datasets to demonstrate their broad applicability.

Key Insights and Ideas

  • The "Low-Dimensional Representation" Hypothesis: The most significant insight is the observation that even complex ML models often produce predictions that have a surprisingly simple, often linear, relationship with the true outcome. This makes correction feasible without needing to understand the internal workings of every ML algorithm.
  • Decoupling Prediction and Inference: The method elegantly separates the task of prediction (using complex ML) from the task of inference (using simpler statistical models), allowing each to be optimized independently while still accounting for their interaction.
  • Leveraging the Testing Set: The testing set, often used for hyperparameter tuning or model selection, is repurposed here to estimate the crucial relationship between observed and predicted outcomes, providing the key to correction.
  • Generalizability: The postpi approach is designed to be model-agnostic, meaning it can be applied to predictions from any ML algorithm, from simple linear models to complex deep neural networks.

Key Results

The paper demonstrates that the postpi approach significantly improves statistical inference compared to the "no correction" baseline:

  • Bias Reduction: Postpi methods consistently reduce bias in coefficient estimates across simulated and real-world applications. For instance, in the binary simulation, the RMSE of estimates was reduced from 2.94 to 0.53.
  • Improved Standard Error Estimation: Standard errors are corrected to be more accurate, moving away from the overly small values produced by uncorrected inference. In the continuous simulation, the RMSE for SEs dropped from 0.088 to 0.015.
  • More Reliable Hypothesis Testing: Corrected t-statistics and p-values lead to better control of Type I error rates. In the binary simulation, the RMSE for t-statistics was reduced from 2.06 to 2.04 (with a slight conservative bias, which is acceptable).
  • Practical Applications: The methods showed substantial improvements in inferring associations between gene expression and tissue types, and in describing cause-of-death distributions. For example, in the tissue type prediction, the RMSE of estimates was reduced from 0.36 to 0.08.

Limitations and Future Work

  • Assumption of Data Distribution: Both methods assume that the training, testing, and validation sets originate from the same data-generating distribution. If this assumption is violated (e.g., due to dataset shift), the corrections may not hold.
  • Fitness of the Relationship Model: The effectiveness of postpi relies on the ability of the chosen relationship model (e.g., linear regression) to accurately capture the link between observed and predicted outcomes. If this relationship is very weak or complex, the correction might be less effective.
  • Covariate Subset: The current methods assume that the covariates used in the downstream inferential model are a subset of those used in the prediction model. Introducing new covariates not seen during prediction training could pose challenges.
  • Future Work: The authors suggest exploring methods to handle violations of the data distribution assumption, such as using data normalization techniques. They also highlight the potential for extending the approach to more complex inferential models beyond generalized linear models. The availability of an open-source R package facilitates wider adoption and further development.

Summary updated

Abstract

Many modern problems in medicine and public health leverage machine-learning methods to predict outcomes based on observable covariates. In a wide array of settings, predicted outcomes are used in subsequent statistical analysis, often without accounting for the distinction between observed and predicted outcomes. We call inference with predicted outcomes postprediction inference. In this paper, we develop methods for correcting statistical inference using outcomes predicted with arbitrarily complicated machine-learning models including random forests and deep neural nets. Rather than trying to derive the correction from first principles for each machine-learning algorithm, we observe that there is typically a low-dimensional and easily modeled representation of the relationship between the observed and predicted outcomes. We build an approach for postprediction inference that naturally fits into the standard machine-learning framework where the data are divided into training, testing, and validation sets. We train the prediction model in the training set, estimate the relationship between the observed and predicted outcomes in the testing set, and use that relationship to correct subsequent inference in the validation set. We show our postprediction inference (postpi) approach can correct bias and improve variance estimation and subsequent statistical inference with predicted outcomes. To show the broad range of applicability of our approach, we show postpi can improve inference in two distinct fields: modeling predicted phenotypes in repurposed gene expression data and modeling predicted causes of death in verbal autopsy data. Our method is available through an open-source R package: https://github.com/leekgroup/postpi.
BibTeX
@inproceedings{wang2020methods,
  author = {Siruo Wang and Tyler H. McCormick and Jeffrey T. Leek},
  title = {{Methods for correcting inference based on outcomes predicted by machine learning}},
  booktitle = {Proceedings of the National Academy of Sciences},
  year = {2020},
  doi = {10.1073/pnas.2001238117},
}
Figure