Learning the Ranking of Causal Effects with Confounded Data

Carlos Fernández-Loría, Jorge Loría

Abstract

Decision makers often want to identify the individuals for whom some intervention or treatment will be most effective in order to decide who to treat. In such cases, decision makers would ideally like to rank potential recipients of the treatment according to their individual causal effects. However, the available data may be completely inadequate to estimate causal effects accurately. We formalize a new assumption -- the rank preservation assumption (RPA) -- that defines when data are suitable to learn how to rank individuals according to their causal effects, even if the effects themselves cannot be accurately estimated. The RPA holds when there is data to estimate a scoring variable that induces the same ranking of individuals as the causal effect of interest. Some of the scoring variables we consider are confounded estimates, proxy causal effects, and non-causal quantities. We show that such scoring variables can work well for treatment assignment if the RPA is met, and potentially even better than using causal effects as scores. We also show that the RPA holds under conditions that are more general and weaker than the typical assumptions made in observational studies. Finally, we showcase how practitioners can apply and evaluate alternative scoring models (including non-causal models) to maximize the causal impact of their targeting decisions.

Introduction

In fields like marketing and public policy, the primary goal is often treatment prioritization: deciding which individuals to target with an intervention to maximize impact. The standard approach involves estimating the Conditional Average Treatment Effect (CATE)—the specific "lift" an individual receives from a treatment. However, CATE estimation is notoriously difficult because it requires counterfactual data, which is often noisy or unavailable. This paper explores why simple predictive "proxies" (like baseline purchase probability) often outperform sophisticated causal models in real-world decision-making.

Theoretical Framework: Causal Ordering and RPA

The authors introduce the concept of Causal Ordering, focusing on the task of ranking individuals by their treatment effects rather than estimating the exact effect size. They formalize this through the Rank Preservation Assumption (RPA). The RPA holds if a scoring variable induces the same ranking as the true causal effect. To support this framework, the paper defines several key terms and assumptions necessary for evaluating these proxies.

Table 1: Key terms and definitions for causal ranking and the Rank Preservation Assumption.

A critical component of this theory is Dominant Moderation. This occurs when a latent variable $\theta$ explains the majority of the variation in the treatment effect $\beta$. If a signal variable $S$ is also driven by $\theta$, it can serve as an effective proxy for ranking, even if $S$ itself is not a causal measure of the treatment's impact.

Figure 1: Causal diagram illustrating how a latent variable $\theta$ moderates the treatment effect $\beta$ and influences the signal variable $S$.

The SNR-Alignment Trade-off

The paper’s core practical contribution is the SNR-Alignment Trade-off. A model's utility for ranking depends on its alignment with the true effect and its Signal-to-Noise Ratio (SNR):

  • CATE models have high theoretical alignment but often suffer from low SNR because they are hard to estimate precisely.
  • Proxies may have imperfect alignment but often boast much higher SNR because they target outcomes that are easier to predict (e.g., "did they buy?" vs. "did the ad cause them to buy?").

Systematic bias in a proxy can actually be beneficial for ranking if it increases with the effect size, as it helps distinguish high-effect individuals from low-effect ones despite random noise.

Empirical Validation

The framework was validated using the Criteo Uplift dataset, a randomized advertising experiment with 14 million users. In this high-stakes environment, conversions were rare (0.3%), making direct CATE estimation extremely noisy.

The results showed that a simple non-causal "Conversion Rate" proxy outperformed the CATE-on-conversions model in both Area Under the Qini Curve (AUQC) and Top 10% Uplift. The authors' diagnostic tools correctly anticipated this, showing that the proxy's 15x higher SNR more than compensated for its imperfect alignment with the true causal lift.

Conclusion

This research demonstrates that when resources are limited, precision in ranking is often more valuable than accuracy in estimation. By identifying the conditions under which non-causal proxies align with causal effects, practitioners can make better targeting decisions using models that are easier to estimate and more robust to noise.

Summary updated

Abstract

Decision makers often want to identify the individuals for whom some intervention or treatment will be most effective in order to decide who to treat. In such cases, decision makers would ideally like to rank potential recipients of the treatment according to their individual causal effects. However, the available data may be completely inadequate to estimate causal effects accurately. We formalize a new assumption -- the rank preservation assumption (RPA) -- that defines when data are suitable to learn how to rank individuals according to their causal effects, even if the effects themselves cannot be accurately estimated. The RPA holds when there is data to estimate a scoring variable that induces the same ranking of individuals as the causal effect of interest. Some of the scoring variables we consider are confounded estimates, proxy causal effects, and non-causal quantities. We show that such scoring variables can work well for treatment assignment if the RPA is met, and potentially even better than using causal effects as scores. We also show that the RPA holds under conditions that are more general and weaker than the typical assumptions made in observational studies. Finally, we showcase how practitioners can apply and evaluate alternative scoring models (including non-causal models) to maximize the causal impact of their targeting decisions.
BibTeX
@misc{fernandezloria2022learning,
  author = {Carlos Fernández-Loría and Jorge Loría},
  title = {{Learning the Ranking of Causal Effects with Confounded Data}},
  year = {2022},
  eprint = {2206.12532},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2206.12532},
}
Figure