Abstract
Algorithms make a growing portion of policy and business decisions. We develop a treatment-effect estimator using algorithmic decisions as instruments for a class of stochastic and deterministic algorithms. Our estimator is consistent and asymptotically normal for well-defined causal effects. A special case of our setup is multidimensional regression discontinuity designs with complex boundaries. We apply our estimator to evaluate the Coronavirus Aid, Relief, and Economic Security (CARES) Act, which allocated many billions of dollars worth of relief funding to hospitals via an algorithmic rule. The funding is shown to have little effect on COVID-19-related hospital activities. Naive estimates exhibit selection bias.
Introduction
We have entered an era of "algorithmic governance." From medical diagnoses and loan approvals to Medicaid eligibility and emergency relief distribution, automated systems increasingly determine the allocation of resources and opportunities. While these "black box" systems are often criticized for their complexity, they offer a unique opportunity for researchers. Because an algorithm makes decisions based strictly on observable data, its recommendations are "conditionally random." This paper argues that these algorithms are, in effect, natural experiments.
However, measuring the causal impact of algorithmic decisions is technically challenging. Traditional tools like Regression Discontinuity Design (RDD) are effective for simple, one-dimensional rules (e.g., a single age cutoff). Modern algorithms, however, often involve dozens of variables and complex, non-linear logic. This high dimensionality creates a "curse of dimensionality" that standard methods cannot navigate. Furthermore, many algorithms are deterministic, meaning the probability of a recommendation is binary (0 or 1), which complicates traditional propensity score methods.
Methodology: The Approximate Propensity Score
To address these challenges, we introduce a unified framework for causal inference based on the Approximate Propensity Score (APS).
The APS Concept
Unlike a standard propensity score, which represents the probability of treatment given a set of covariates, the APS is defined as the average probability of a treatment recommendation within a tiny, shrinking neighborhood ($\delta$) around a specific data point. By focusing on this local neighborhood, the APS "smooths out" sharp algorithmic boundaries, allowing researchers to treat the decision as locally random.
Simulation-Based Estimation
For complex algorithms where the decision boundary is not mathematically tractable, we propose a simulation-based approach. For every individual in the dataset, we slightly perturb their input variables multiple times and observe how the algorithm’s recommendation changes. The average of these simulated outcomes serves as the APS. This "brute force" method allows the estimator to function even when the underlying algorithm is a complex machine learning model, such as a Random Forest or a Neural Network.
Two-Stage Least Squares (2SLS)
The APS is then integrated into a Two-Stage Least Squares (2SLS) framework. We use the algorithm’s recommendation as an Instrumental Variable (IV), controlling for the APS. This setup isolates the variation in treatment driven solely by the algorithm's local logic, effectively mimicking a randomized controlled trial at the decision boundary.
Empirical Application: The CARES Act
We demonstrate the utility of the APS estimator by evaluating the Coronavirus Aid, Relief, and Economic Security (CARES) Act. This legislation allocated $175 billion in relief funding to healthcare providers. The allocation followed specific algorithmic rules based on 2018 financial data.
To conduct this analysis, we utilized the Healthcare Cost Report Information System (HCRIS) for hospital financial data and the COVID-19 Reported Patient Impact and Hospital Capacity dataset to track hospital activity. The goal was to determine if the billions of dollars in funding actually increased hospital capacity or activity related to COVID-19.
Key Results
Simulation Accuracy
In Monte Carlo simulations involving a mix of randomized experiments and deterministic machine learning decisions across 100 covariates, the APS estimator significantly outperformed standard Ordinary Least Squares (OLS) and traditional 2SLS models. It demonstrated lower Root Mean Squared Error (RMSE) and provided more accurate coverage of true causal effects, proving robust against the curse of dimensionality.
The CARES Act "Null" Finding
The real-world application yielded striking results. A naive OLS analysis suggested a strong positive correlation: every $1 million in funding was associated with approximately 5.6 additional COVID-19 patients. However, this correlation is likely driven by selection bias—hospitals that were already seeing more patients were more likely to receive higher funding.
When applying the APS estimator to control for this bias, the estimated effect dropped to near zero and became statistically insignificant. This suggests that while the CARES Act provided necessary financial support to busy hospitals, the funding itself did not materially increase the volume of COVID-19-related hospital activities beyond what would have occurred otherwise.
Discussion and Limitations
The APS method reveals that many policy interventions may be "poorly targeted" if the objective is to change behavior rather than simply provide a transfer of wealth. By identifying the causal impact at the boundary, we can better understand the marginal utility of algorithmic decisions.
However, the method has limitations:
- Bandwidth Sensitivity: The choice of the neighborhood size ($\delta$) is critical. While our results are robust across various choices, a universal data-driven method for selecting the optimal bandwidth in high-dimensional settings is still needed.
- Sample Size: The estimator relies on large-sample asymptotics for its statistical properties, and its performance in small datasets requires further investigation.
Conclusion
Algorithms are more than just decision-making tools; they are structured experiments. By using the Approximate Propensity Score, researchers can unlock the causal insights hidden within complex, high-dimensional systems. As organizations continue to automate governance, the ability to audit these decisions for their true impact becomes essential for both effective policy design and corporate accountability. Future work should focus on "optimal policy learning"—using these causal estimates to refine algorithms to achieve better societal outcomes.
Summary updated