TL;DR
When choosing the best policy based on data, simply "picking the winner" with the highest estimate (Empirical Welfare Maximization) often ignores the risk of estimation error. This paper introduces PoLeCe (Policy Learning with Confidence), a decision rule that balances estimated performance against uncertainty to provide a "reporting guarantee": a mathematically backed promise that the chosen policy’s actual welfare will exceed a specific threshold with high confidence.
What Problem Does This Paper Address?
Imagine a policymaker deciding how to allocate a limited budget between two social programs. Program A has a high estimated "bang for the buck," but the estimate comes from a small, noisy study. Program B has a slightly lower estimate but was studied extensively with very high precision.
Standard statistical decision theory often relies on Empirical Welfare Maximization (EWM)—essentially a "plug-in" rule that treats estimates as ground truth. However, EWM is prone to the "winner's curse": it tends to select policies that had "lucky" positive estimation errors. The authors argue that decision-makers are often risk-averse and need a way to hedge against this uncertainty. The challenge is creating a rule that isn't just "cautious" but provides a formal guarantee on the minimum level of welfare the chosen policy will deliver.
Methods
The authors propose a class of Risk-Aware (RW) policy rules. These rules don't just maximize estimated welfare $V(\pi)$; they maximize a risk-adjusted objective: $$\text{Objective} = \text{Estimated Welfare} - (k \times \text{Estimation Risk})$$
The core contribution is the PoLeCe rule, which specifically calibrates the penalty term $k$.
- The Efficient Decision Frontier: By varying $k$, the authors trace a frontier of policies that offer the maximum possible welfare for any given level of estimation risk.
- The Bootstrap Quantile: PoLeCe sets $k$ using a Gaussian bootstrap to approximate the quantile of the maximal estimation error.
- The Reporting Guarantee: By choosing $k$ this way, the objective function itself becomes a Lower Confidence Bound (LCB). If the PoLeCe objective value is 1.5, the policymaker can state with 95% confidence that the true welfare of the implemented policy is at least 1.5.
- Computational Efficiency: To make this practical for complex allocations (like splitting a budget across many programs), they reformulate the bootstrap calculation as a root-finding problem involving second-order cone programs, which is much faster than traditional grid searches.
Datasets
The paper evaluates the PoLeCe rule using several real-world economic contexts:
- The Policy Impacts Library: A database of Marginal Value of Public Funds (MVPF) estimates for US social programs. The authors focus on 14 programs evaluated via Randomized Control Trials (RCTs).
- Immunization Nudges: Data from a large-scale RCT in India (Banerjee et al., 2025) testing 74 different ways to encourage measles vaccinations.
- Informal Savings Technologies: Data from a field experiment in Kenya (Dupas & Robinson, 2013) on health investment.
- Alcohol and Self-Control: Data from a study on sobriety incentives for rickshaw drivers in India (Schilbach, 2019).
Key Insights and Ideas
- Heterogeneity is a Hedge: The PoLeCe rule performs exceptionally well when there is "risk heterogeneity"—i.e., when some high-performing policies are much more precisely estimated than others. In these cases, PoLeCe can achieve much lower "regret" (the gap between the chosen policy and the theoretical best) than EWM.
- Beyond Winner-Take-All: In budget allocation, EWM is a "corner solution"—it puts 100% of the money into the single program with the highest point estimate. PoLeCe naturally leads to diversification, spreading the budget across multiple high-performing, high-precision programs, similar to a Markowitz portfolio in finance.
- Transparency as a Feature: The authors align their work with the needs of organizations like GiveWell, which have recently emphasized that accounting for uncertainty is vital for "transparency" in philanthropic and governmental decision-making.
Key Results
- Lower Tail Regret: In simulations calibrated to real RCT data, PoLeCe significantly reduced "tail regret" (the worst-case scenarios). For example, in the "Measles shots" simulation, the 95th-percentile regret for EWM was 32.10%, while PoLeCe slashed it to 12.84%.
- Valid Guarantees: The "Average Welfare LCB" for PoLeCe was consistently higher and more reliable than EWM across all tested scenarios, proving that the reporting guarantee holds up in practice.
- Policy Shifts: In the US social program application, EWM suggested putting the entire budget into "Holistic Wrap-around Services." PoLeCe instead recommended a mix dominated by Medicaid for single adults and job training (JTPA), because those programs had slightly lower estimates but much tighter standard errors.
Limitations and Future Work
- Point Identification Assumption: The current framework assumes that the welfare of each policy is "point identified" (we can estimate a single value for it). The authors note that extending this to "partial identification"—where we only know a range for the welfare—is a promising but complex next step.
- Gaussian Dependency: The method relies on the Gaussian approximation (Central Limit Theorem). While standard in econometrics, it may require adjustments for very small sample sizes where the distribution of errors is highly non-normal.
- Static Settings: The paper focuses on a one-time decision. Future research could explore how "Policy Learning with Confidence" evolves in an active learning or "bandit" setting where the DM can collect more data on the fly.
Summary updated