In the 16th century, Albrecht Dürer drew a rhinoceros he had never seen by aggregating descriptions from others—a classic case of statistical inference. Today, scientists are doing something similar: using AI "descriptions" (predictions) to fill in the gaps where real data is too expensive or impossible to collect. This paper explores the burgeoning field of Inference with Predicted Data (IPD) and asks how we can ensure our scientific conclusions remain valid when the "data" we’re analyzing was generated by a black-box algorithm.
TL;DR
As data collection costs rise, researchers are increasingly replacing hard-to-measure outcomes with predictions from pre-trained AI models. This paper formalizes the "Inference with Predicted Data" (IPD) framework, highlighting how to statistically "rectify" these predictions using small sets of ground-truth data to avoid biased results and overconfident conclusions.
What Problem Does This Paper Address?
In many fields, the "ground truth" is becoming a luxury. In genetics, full genome sequencing is expensive, so researchers use "imputed" sequences. In social science, researchers might use street-level imagery and AI to predict a neighborhood's demographic makeup rather than conducting a door-to-door census.
The problem is the "Naive Approach": treating these AI predictions as if they were observed facts. If an algorithm has a slight bias or if the population it was trained on differs from the current study, standard statistical tools will produce "hallucinated" certainty. Specifically, the paper identifies three sources of error:
- The systematic gap between predicted outcomes and reality.
- The uncertainty of the machine learning model itself.
- The failure to propagate these uncertainties into the final scientific conclusion.
Methods
The paper provides a conceptual overview of the IPD paradigm, contrasting it with classical methods like regression imputation and semi-supervised learning.
The core IPD workflow involves:
- The Prediction Rule ($f$): A pre-trained, often "black-box" model (like AlphaFold or ChatGPT) that maps features ($X$) to a predicted outcome ($\hat{Y}$).
- The Labeled Subset: A small, "gold-standard" dataset where both the features and the true outcomes ($Y$) are known.
- The Rectifier ($\Delta$): A statistical correction factor. Instead of just using the predictions, IPD methods use the small labeled set to estimate the bias of the algorithm.
- Rectified Inference: Recent frameworks like Prediction-Powered Inference (PPI) and PPI++ use this "rectifier" to adjust the final estimate (e.g., a regression coefficient). This ensures the final result is unbiased and that the standard errors (the "plus-minus" range) accurately reflect the fact that most of the data was guessed by an AI.
Datasets
As a perspective and review paper, it does not introduce a new benchmark dataset. Instead, it synthesizes a wide array of existing research. Table 1 in the paper serves as a meta-dataset of the IPD landscape, citing studies across:
- Biomedical Studies: Using XGBoost to map gene interactions.
- Public Health: Predicting influenza-like illnesses from Google search queries.
- Sociology: Using computer vision on Google Street View to predict voting patterns.
- Political Science: Using Latent Dirichlet Allocation (LDA) to analyze partisan speeches.
Key Insights and Ideas
- The "Black Box" Constraint: Unlike traditional semi-supervised learning, IPD assumes the researcher cannot retrain the model. They are "downstream" users of a tool (like a Large Language Model) and must account for its quirks without seeing its internal weights or training data.
- Information Silos: IPD is a solution for a world of data privacy. If a hospital has a model but cannot share the patient data used to train it, IPD allows other researchers to use that model safely by validating it on a small, local sample.
- The "No-Harm" Guarantee: Modern IPD methods (like PPI++) provide mathematical guarantees that the resulting inference will be no worse than if the researcher had ignored the AI and only used the tiny labeled dataset.
Key Results
The paper summarizes the evolution of IPD from early "post-prediction inference" (PostPI) to modern "design-based" approaches.
- Bias Correction: IPD successfully shifts estimates from the "naive" (biased) result toward the true population parameter.
- Valid Uncertainty: While the naive approach often produces artificially small p-values (making results look more significant than they are), IPD "inflates" standard errors to a scientifically honest level.
- Efficiency: The paper notes that while IPD is safer than the naive approach, it is also more efficient than the "classic" approach (throwing away the AI predictions entirely), as it leverages the information hidden in the unlabeled features ($X_U$).
Limitations and Future Work
The authors are candid about the "garbage in, garbage out" risk. If a prediction model is exceptionally low-quality, the "rectification" process might introduce so much variance that the benefits of using the predicted data vanish.
Future work is needed to:
- Determine the optimal "tipping point"—at what level of model accuracy does IPD stop being helpful?
- Develop better methods for "black-box" validation when the underlying model assumptions (like linearity) are unknown.
- Bridge the gap between the "Total Survey Error" framework used in sociology and the algorithmic error frameworks used in machine learning.
In short, while we may not "need" perfect data for every single observation, we desperately need the statistical rigor to know when our AI-generated "rhinos" are leading us astray.
Summary updated