Qini Curves for Multi-Armed Treatment Rules

Erik Sverdrup, Han Wu, Susan Athey, Stefan Wager

TL;DR

This paper generalizes the Qini curve—a popular tool for evaluating targeting rules—from simple binary treatments to complex scenarios with multiple treatment arms and varying costs. The authors provide an efficient algorithm to compute these curves and a rigorous statistical framework for building confidence intervals, enabling researchers to quantify the value of "upgrading" treatments for specific subgroups under a budget.

What Problem Does This Paper Address?

In fields like marketing, medicine, and public policy, we often need to decide not just who to treat, but how to treat them. While traditional Qini curves help evaluate the benefit of a single treatment (e.g., "Should we send this coupon?"), they fail when multiple options exist (e.g., "Should we send a 5%, 10%, or 20% coupon?").

The challenge is two-fold:

  1. Cost-Benefit Trade-offs: Different treatments have different costs and different effects on different people. A high-cost drug might be slightly better than a low-cost one for most, but significantly better for a small subgroup.
  2. Budget Constraints: Decisions are rarely made in a vacuum; there is usually a fixed budget. We need to know the optimal mix of treatments to maximize total "gain" (e.g., lives saved, votes gained) at every possible spending level.

Methods

The authors bridge the gap between causal inference and optimization by recasting the multi-armed targeting problem as a linear multiple-choice knapsack problem.

  • The Convex Hull: For any individual, the only treatment arms worth considering are those that lie on the "convex hull" of the cost-reward plane. If an arm is more expensive and less effective than a combination of others, it’s discarded.
  • Incremental Efficiency: The optimal policy is characterized by "incremental cost-benefit ratios." As the budget increases, the algorithm selects the next "unit-arm" pair with the highest marginal return on investment.
  • Efficient Algorithm: They developed a path-following algorithm that computes the entire Qini curve in log-linear time ($O(nK \log nK)$). This is significantly faster than solving individual linear programs for every budget level.
  • Statistical Inference: The paper's heavy lifting is a Central Limit Theorem for the estimated policy values. They use Augmented Inverse Propensity Weighting (AIPW) to robustly estimate gains and a half-sampling bootstrap method to generate valid confidence intervals.

Datasets

The authors validated their approach using:

  1. Synthetic Data: A three-armed data generating process (adapted from recent literature) designed to test the coverage of their confidence intervals across different sample sizes (1,000 to 10,000 units).
  2. Election Turnout Data: A real-world dataset from a large-scale field experiment by Gerber et al. (2008). This study involved four different mailing treatments (from "Civic Duty" reminders to the more intrusive "Neighbors" mailing) sent to 180,002 households to encourage voting.

Key Insights and Ideas

  • The "Upgrade" Logic: Unlike binary Qini curves that just add more people to the "treated" pool, the multi-armed curve accounts for "upgrading" individuals from a cheaper, less effective treatment to a costlier, more effective one as the budget grows.
  • Budget Flexibility vs. Targeting: In the election turnout application, the authors found that the gains from using multiple arms weren't just about knowing who to target with what, but also about the flexibility to split the budget between a cheap-but-weak arm and an expensive-but-strong arm.
  • Statistical Rigor: By providing exact large-sample confidence intervals, the authors allow practitioners to perform hypothesis tests. You can now statistically prove whether adding a third treatment arm actually provides a significant benefit over just using two.

Key Results

  • Computational Speed: The implementation is highly efficient; it can compute the full solution path for a dataset of one million units and five treatment arms in about 1.5 seconds on a standard laptop.
  • Nominal Coverage: In simulations, their 95% bootstrap confidence intervals consistently achieved near 95% coverage, proving the reliability of their statistical framework.
  • Real-World Impact: In the election turnout study, they showed that at a specific budget level, an optimal multi-armed policy could increase turnout by 2%, whereas the best single-arm policy only achieved 1%.

Limitations and Future Work

  • Area Under the Curve (AUC): While the paper provides point-wise inference (confidence intervals for specific budget points), it does not yet provide a functional central limit theorem for the "Area Under the Qini Curve" (AUQC) as a single summary statistic. The authors flag this as a promising direction for future research.
  • Known Costs: The current framework assumes that the costs of treatments are known or can be modeled as a fixed function of covariates. Scenarios where costs themselves are unknown and must be estimated from data are not the primary focus.
  • Software Availability: To encourage adoption, the authors released an open-source implementation called maq for both R and Python.

Summary updated

Abstract

Qini curves have emerged as an attractive and popular approach for evaluating the benefit of data-driven targeting rules for treatment allocation. We propose a generalization of the Qini curve to multiple costly treatment arms that quantifies the value of optimally selecting among both units and treatment arms at different budget levels. We develop an efficient algorithm for computing these curves and propose bootstrap-based confidence intervals that are exact in large samples for any point on the curve. These confidence intervals can be used to conduct hypothesis tests comparing the value of treatment targeting using an optimal combination of arms with using just a subset of arms, or with a non-targeting assignment rule ignoring covariates, at different budget levels. We demonstrate the statistical performance in a simulation experiment and an application to treatment targeting for election turnout.
BibTeX
@article{sverdrup2024qini,
  author = {Erik Sverdrup and Han Wu and Susan Athey and Stefan Wager},
  title = {{Qini Curves for Multi-Armed Treatment Rules}},
  journal = {Journal of Computational and Graphical Statistics},
  year = {2024},
  doi = {10.1080/10618600.2024.2418820},
}

Loading related content...

Figure