TL;DR
This paper introduces the "amenability framework" to address the challenge of prioritizing individuals for interventions when estimating direct causal effects is difficult or impossible. It proposes that focusing on an individual's "amenability"—their latent proclivity to be influenced—can lead to more effective targeting than traditional methods, even outperforming causal effect estimation in certain scenarios.
What Problem Does This Paper Address?
In many real-world applications like advertising, customer retention, and behavioral nudging, organizations need to decide who to target with limited resources. The ideal approach would be to estimate the causal effect of an intervention for each individual and target those with the highest potential benefit. However, directly estimating these individual-level causal effects is often infeasible due to:
- Data limitations: Interventions might be new, or outcomes might be too slow or rare to measure reliably.
- Cost and complexity: Running large-scale experiments to estimate individual effects can be prohibitively expensive or impractical.
- Noisy signals: Even with data, intervention effects can be small relative to noise, making precise estimation challenging.
This leads to a common practice of using predictive models (e.g., predicting purchase probability) to guide targeting. However, research has questioned whether these predictive scores, which don't directly estimate causal effects, are truly effective. This paper bridges this gap by investigating when and why these predictive scores can effectively rank individuals for intervention, even without estimating causal effects.
Methods
The core of the paper is the Amenability Framework. Instead of focusing on estimating the Conditional Average Treatment Effect (CATE), it shifts the focus to amenability, defined as an individual's latent proclivity to be influenced by an intervention.
Key concepts and methods include:
- Amenability Dominance Assumption: This assumption states that if an intervention has a larger effect on individual i than individual j, it's because i is more amenable than j.
- Conditional Average Signal (CAS): This is what predictive models estimate (e.g., baseline outcome, surrogate effect). The paper explores conditions under which CAS can serve as a proxy for amenability.
- Unbiased Ordering (UO): A strict condition where the ranking produced by the CAS is identical to the ranking by the CATE.
- Unbiased Classification (UC): A looser condition where the top k individuals ranked by CAS are the same as the top k by CATE.
- Amenability Mediation & Monotonicity: These are theoretical conditions that, when met, guarantee unbiased ordering. Amenability Mediation requires the signal to be a "pure" signal of amenability, while Amenability Monotonicity requires a consistent directional relationship between the signal and amenability.
- Signal Strength vs. Structural Alignment: The paper highlights a trade-off: a model might have strong structural alignment with the true causal effect but be noisy, or it might have weaker alignment but a stronger, less noisy signal of amenability.
Datasets
The paper uses a large-scale randomized experiment dataset from Criteo, an advertising platform. This dataset includes:
- Nearly 14 million observations.
- 11 user features.
- Treatment indicator for ad exposure.
- Two outcome variables: website visits and conversions.
This dataset is used to benchmark different models for causal ordering and classification.
Key Insights and Ideas
- Amenability is Key: The paper argues that for effective targeting, understanding who is amenable to an intervention is more crucial than precisely estimating how much the intervention will affect them.
- Predictive Scores Can Outperform Causal Estimates: Counter-intuitively, well-estimated predictive scores (like baseline conversion rates) can lead to better rankings for interventions than noisy estimates of true causal effects, especially when the scores capture amenability better.
- Bias Can Be Beneficial: While bias is generally undesirable for estimating effects, it can actually improve ranking accuracy if it systematically separates individuals with high vs. low effects, offsetting variance-driven errors.
- Surrogate Effects as Causal Signals: The usefulness of a surrogate effect for targeting isn't about its position on the causal pathway, but its correlation with amenability.
- Practicality Over Perfect Alignment: In resource-constrained settings, a reliable, albeit imperfect, proxy for amenability can be more valuable for prioritization than a theoretically ideal but difficult-to-estimate causal effect.
Key Results
- Empirical Evidence: In a large-scale advertising dataset, a simple non-causal model predicting baseline conversion rates outperformed a model directly estimating causal effects (CATE) in both causal ordering (ranking individuals by effect size) and causal classification (identifying top responders).
- Conversion Rate Model Success: The conversion rate model, which predicts conversion probability without an ad, achieved better ranking and classification performance than CATE models, even when the CATE model was trained on 10 million observations. This was attributed to the positive correlation between bias and treatment effect, which improved causal ordering.
- Theoretical Conditions: The paper provides formal conditions (Amenability Mediation and Monotonicity) under which predictive scores are guaranteed to provide unbiased ordering. It also shows that even when these strict conditions fail, scores can still be useful for classification.
Limitations and Future Work
- Single Latent Trait Assumption: The framework is best suited for settings where a single latent trait drives heterogeneity in intervention effects. When multiple uncorrelated traits contribute to variation, or when scores violate monotonicity, predictive models may become misleading.
- Untestable Assumptions: The structural assumptions underlying the framework are not directly testable without experimental data, relying instead on theory, domain knowledge, or indirect validation.
- Future Directions: The authors suggest exploring:
- Domains where modeling amenability is more effective than estimating causal effects.
- Data-generating processes that support amenability inference.
- Methods to directly estimate amenability rather than effects.
- Embedding amenability-based ranking into richer decision-making frameworks that optimize downstream outcomes or incorporate constraints.
Summary updated