Abstract: Machine learning is increasingly used to select which individuals receive limited-resource interventions in domains such as human services, education, development, and more. However, it is often not apparent what the right quantity is for models to predict. Policymakers rarely have access to data from a randomized controlled trial (RCT) that would enable accurate estimates of which individuals would benefit more from the intervention, while observational data creates a substantial risk of bias in treatment effect estimates. Practitioners instead commonly use a technique termed "risk-based targeting" where the model is just used to predict each individual's status quo outcome (an easier, non-causal task). Those with higher predicted risk are offered treatment. There is currently almost no empirical evidence to inform which choices lead to the most effective machine learning-informed targeting strategies in social domains. In this work, we use data from 5 real-world RCTs in a variety of domains to empirically assess such choices. We find that when treatment effects can be estimated with high accuracy (which we simulate by allowing the model to partially observe outcomes in advance), treatment effect based targeting substantially outperforms risk-based targeting, even when treatment effect estimates are biased. Moreover, these results hold even when the policymaker has strong normative preferences for assisting higher-risk individuals. However, the features and data actually available in most RCTs we examine do not suffice for accurate estimates of heterogeneous treatment effects. Our results suggest treatment effect targeting has significant potential benefits, but realizing these benefits requires improvements to data collection and model training beyond what is currently common in practice.
Introduction
Policymakers frequently face a "triage" problem: they must allocate a limited budget—such as a fixed number of slots in a job training program or a specific quantity of medical supplies—across a large population in need. Traditionally, this is addressed through risk-based targeting, which prioritizes individuals with the worst predicted "status quo" outcomes. The underlying logic is intuitive: those who are worst off should be the primary recipients of aid.
However, from a causal inference perspective, this approach may be inefficient. A high-risk individual might be "treatment-resistant," while a moderate-risk individual might experience a substantial improvement from the same intervention. Treatment-effect-based targeting seeks to maximize social welfare by prioritizing those with the highest Heterogeneous Treatment Effect (HTE)—the specific "lift" an individual receives from the resource. This paper investigates the conditions under which policymakers should transition from simple risk-based models to more complex causal models.
Methodology
The study compares targeting strategies using the Neyman-Rubin Potential Outcomes framework. To address the challenge of missing counterfactual data (the fact that we cannot observe both treated and untreated outcomes for the same individual), the researchers employed Doubly-Robust (DR) Estimators to generate "pseudo-outcomes" for evaluation.
To test the robustness of these models against real-world imperfections, the authors systematically introduced simulated confounding into their datasets. By removing specific units, they simulated adverse selection—where individuals who would have benefited most are missing from the training data—thereby creating biased estimates. Finally, policies were evaluated using diverse Social Welfare Functions, including Weighted Utilitarian functions (prioritizing the at-risk) and Nash Welfare (balancing total impact with equity).
Empirical Datasets
The study utilizes five diverse real-world RCT datasets to ensure generalizability:
- Targeting the Ultra Poor (TUP): Capital grants for low-income families in India.
- NSW (National Supported Work): A classic job training program dataset.
- Postoperative Pain: Testing a licorice solution to reduce throat pain after surgery.
- Acupuncture: Evaluating therapy for chronic headaches.
- Tennessee STAR: A large-scale study on the effects of smaller class sizes on student performance.
Key Findings
The Risk-Benefit Myth
A central finding of this research is the decoupling of baseline risk and treatment benefit. In many policy contexts, it is assumed that those at the highest risk will derive the greatest benefit from an intervention. However, the researchers found a noisy and inconsistent relationship between baseline risk and actual treatment effect across all five domains.
As illustrated in the figure above, the smoothed DR predictions for treatment effects do not follow a uniform linear trend relative to baseline risk percentiles. In the NSW dataset, the effect is relatively flat; in the Ultra Poor dataset, it peaks around the 75th percentile; and in the STAR dataset, it shows significant fluctuations. This suggests that targeting the most vulnerable is often a poor proxy for maximizing the total benefit of a program.
Resilience to Confounding
Surprisingly, even when causal models were trained on heavily confounded (biased) data, they frequently outperformed risk-based targeting. In the STAR (education) and NSW (labor) datasets, treatment-effect targeting remained superior even when up to 20-30% of the most successful cases were systematically removed from the training data. This suggests that a "slightly wrong" causal model is often more effective than a "perfect" non-causal risk model for welfare maximization.
The Data Bottleneck
The primary obstacle to effective treatment-effect targeting is not bias or ethical preference, but rather a lack of high-quality data. The features (covariates) typically collected in standard RCTs are often too sparse to capture the complex patterns required to predict who will benefit most. To justify risk-based targeting over treatment-effect targeting on purely utilitarian grounds, a policymaker would need to value a high-risk individual's welfare approximately 4.5 times more than a lower-risk individual's.
Conclusion
While treatment-effect targeting offers significant potential to increase social welfare, realizing these gains requires a shift in how data is collected and utilized. The current reliance on risk-based "triage" is often a symptom of data limitations rather than optimal policy. Future work must focus on collecting richer administrative datasets and developing models capable of extracting subtle signals of individual benefit to move beyond simple risk-based allocation.
Summary updated