Qini curve estimation under clustered network interference

Rickard K.a. Karlsson, Bram van den Akker, Felipe Moraes, Hugo Manuel Proenca, Jesse H. Krijthe

TL;DR

This paper tackles the critical problem of estimating Qini curves, a tool for evaluating cost-effective treatment policies, when there's "interference" between individuals (e.g., one person's treatment affects another's outcome). They introduce new methods to accurately estimate these curves in the presence of clustered network interference and provide a simulator to test them, offering practical guidance for real-world applications.

What Problem Does This Paper Address?

In many real-world scenarios, like public policy or marketing, treating one individual can influence the outcomes of others. This phenomenon is known as "interference." Standard methods for estimating Qini curves, which help decide how to allocate resources for treatments to maximize benefit within a budget, assume no such interference. When interference is present but ignored, these Qini curves can be significantly misleading, leading to suboptimal or even counterproductive policy decisions. This paper specifically addresses the challenge of estimating Qini curves when interference occurs in "clusters," meaning individuals within a cluster affect each other, but clusters are independent.

Methods

The authors propose a framework for estimating Qini curves under clustered network interference. They introduce three distinct estimation strategies:

  • Standard IPW (Inverse Probability Weighting): An extension of traditional methods, this approach is unbiased but can suffer from high variance, especially with larger clusters.
  • Fractional IPW: This method uses a "fractional exposure mapping" that groups treatment configurations, reducing the complexity of the problem and leading to more favorable variance scaling with cluster size, though it requires stronger assumptions.
  • β-additive IPW: This strategy models interference using a polynomial function of treatment interactions within a cluster. By adjusting the parameter β, it offers a tunable bias-variance trade-off, with smaller β values leading to more restrictive assumptions but potentially lower variance.

They also introduce an "augmented" version of these estimators that incorporates an auxiliary outcome model to further reduce variance without introducing bias.

Datasets

The authors developed a custom marketplace simulator to generate data for their experiments. This simulator mimics an e-commerce environment where clusters represent potential buyers and treatment units are items shown to them. Interference arises through "cannibalization," where a treatment on one item might shift a buyer's purchase from another item rather than increasing overall purchases. The simulator allows for various interference structures, including those that violate the assumptions of the proposed methods, enabling a thorough evaluation.

Key Insights and Ideas

  • Interference is a major bias source: Ignoring clustered network interference can lead to severely biased Qini curves, making it difficult to correctly assess treatment policy effectiveness.
  • Bias-Variance Trade-off is Crucial: Different estimation strategies offer varying balances between bias and variance. The "best" strategy depends on factors like cluster size, data availability, and the acceptable level of bias for the specific application.
  • β-additive IPW offers flexibility: This method provides a tunable parameter (β) that allows practitioners to adjust the strength of interference modeling, offering a practical way to manage the bias-variance trade-off.
  • Augmentation can improve efficiency: Adding an auxiliary outcome model to weighted estimators can reduce variance without compromising unbiasedness, particularly beneficial when initial variance is high.

Key Results

  • The naive estimator (ignoring interference) consistently showed significant bias across various interference structures.
  • The standard IPW estimator was unbiased but exhibited high variance, which increased exponentially with cluster size.
  • The fractional IPW estimator showed improvement over standard IPW in terms of variance scaling but still had limitations.
  • The β-additive IPW estimators (especially with β=1 or β=2) generally performed best, offering lower bias and variance compared to the naive and standard IPW methods. They showed sub-exponential scaling of variance with cluster size.
  • In terms of ranking policies, β-IPW with β=1 achieved perfect rankings in most scenarios, demonstrating its effectiveness in distinguishing between good and bad policies.
  • Augmented estimators showed a reduction in variance compared to their non-augmented counterparts, particularly for the standard IPW estimator.

Limitations and Future Work

  • Untestable Assumptions: The assumptions underlying the fractional and β-additive IPW estimators (Assumptions 2 and 3) are untestable, making it difficult to definitively choose between them without domain expertise.
  • Data-Adaptive Strategies: The authors suggest exploring data-adaptive strategies to select the optimal β for the β-IPW estimator by minimizing an estimated MSE. This approach could provide more robust performance but might introduce its own variance challenges.
  • Generalizing Interference: While the paper focuses on clustered network interference, future work could explore methods for more general interference structures.
  • Practical Implementation: While the proposed methods are theoretically sound, their practical implementation might still require careful consideration of computational complexity and the need for domain expertise.

Summary updated

Abstract

Qini curves are a widely used tool for assessing treatment policies under allocation constraints as they visualize the incremental gain of a new treatment policy versus the cost of its implementation. Standard Qini curve estimation assumes no interference between units: that is, that treating one unit does not influence the outcome of any other unit. In many real-life applications such as public policy or marketing, however, the presence of interference is common. Ignoring interference in these scenarios can lead to systematically biased Qini curves that over- or under-estimate a treatment policy's cost-effectiveness. In this paper, we address the problem of Qini curve estimation under clustered network interference, where interfering units form independent clusters. We propose a formal description of the problem setting with an experimental study design under which we can account for clustered network interference. Within this framework, we introduce three different estimation strategies suited for different conditions. Moreover, we introduce a marketplace simulator that emulates clustered network interference in a typical e-commerce setting. From both theoretical and empirical insights, we provide recommendations in choosing the best estimation strategy by identifying an inherent bias-variance trade-off among the estimation strategies.
BibTeX
@misc{karlsson2025qini,
  author = {Rickard K.A. Karlsson and Bram van den Akker and Felipe Moraes and Hugo Manuel Proenca and Jesse H. Krijthe},
  title = {{Qini curve estimation under clustered network interference}},
  year = {2025},
  eprint = {2502.20097},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2502.20097},
}
Figure