TL;DR
This paper tackles the challenge of using user sentiment from optional surveys as a reliable metric in A/B tests. They propose a novel framework that combines missing data and causal inference techniques to address biases arising from non-response, enabling more accurate estimation of treatment effects on user sentiment.
What Problem Does This Paper Address?
In the world of tech companies, A/B testing is a standard practice for evaluating product changes. While engagement metrics are common, many companies also want to measure user sentiment, which is often collected through optional surveys. The core problem is that these surveys have low response rates, and the users who do respond might be systematically different from those who don't. This "non-response bias" can lead to misleading conclusions when trying to understand the true impact of a change on overall user sentiment. The paper aims to develop methods to accurately estimate the effect of A/B test variations on user sentiment, even when survey responses are incomplete and potentially biased.
Methods
The authors frame the problem as a combination of missing data and causal inference. They propose two main estimation goals:
- Average Treatment Effect (ATE): Estimating the overall impact of a treatment on user sentiment across the entire population. They show this can be broken down into two independent "missing data" problems, one for the treatment group and one for the control group, allowing for parallel computation.
- Average Treatment Effect on Treated and Responded (ATETR): Estimating the impact of a treatment specifically on users who were exposed to the treatment and responded to the survey. This is framed as a more standard causal inference problem, where they leverage techniques for estimating the treatment effect on the treated.
They introduce two key assumptions:
- Assumption 1 (Missing At Random - MAR): Assumes that the reasons for missing survey data can be explained by observed variables.
- Assumption 2: Focuses on the control group and assumes that response behavior is independent of sentiment, given observed confounders.
They then evaluate several estimators, including Outcome Regression (OR), Inverse Propensity Weighting (IPW), Doubly Robust (DR), and Adversarial Balancing (AB), comparing their performance under different scenarios of confounder observability.
Datasets
The paper uses a simulated dataset for its evaluation. The data generation process is described in detail in Appendix B, where latent variables are used to generate user sentiment and response behavior under treatment and control conditions. They specifically design the simulation to satisfy their proposed Assumptions 1 and 2, and to reflect realistic scenarios with differing response rates and potential for bias.
Key Insights and Ideas
- Sentiment as a Metric is Tricky: The paper highlights that while sentiment is a valuable metric, its collection via optional surveys introduces significant challenges due to non-response bias.
- Bridging Missing Data and Causal Inference: The core innovation is framing the problem at the intersection of these two fields, allowing them to adapt existing techniques to the specific context of A/B testing with survey data.
- ATE Decomposition: Breaking down the ATE estimation into two independent missing data problems is a clever way to enable parallelization and potentially more efficient computation.
- Model Misspecification Matters: The simulations reveal that the performance of estimators can degrade significantly when the underlying confounders are not perfectly observed, underscoring the importance of robust methods.
- Simpler Methods Can Be Competitive: In some scenarios, simpler approaches like covariate control (CC) performed surprisingly well, suggesting that overly complex models aren't always necessary.
Key Results
The paper presents simulation results comparing various estimators for both ATE and ATETR.
- ATE Estimation:
- When confounders are fully observed, the naive mean comparison estimator shows the largest bias. Adversarial Balancing (AB) has the best Mean Squared Error (MSE).
- When confounders are noisy (observed as transformations), the performance of Outcome Regression (OR) drops significantly. AB continues to offer a good balance between bias and variance.
- ATETR Estimation:
- When confounders are fully observed, Covariate Control (CC) and Entropy Balancing (EB) show the best MSE performance.
- When confounders are noisy, EB and CC continue to outperform other estimators, with CC providing comparable performance to EB. Inverse Propensity Weighting (IPW) suffers from large variance, and OR performs worse than the naive estimator.
Limitations and Future Work
- Model Misspecification Sensitivity: The authors acknowledge that their methods, like many causal inference techniques, can be sensitive to model misspecification, especially when confounders are not directly observed. This is demonstrated by the performance drop in scenario 2 of their simulations.
- Generalization to More Complex Scenarios: While the paper focuses on a single survey question, they note that their approach can be expanded to sentiment measured by a series of questions.
- Integration with Platforms: The authors suggest their framework, combined with simpler regression or balancing techniques, has the potential to be integrated into existing experimentation platforms for more automated and robust impact calculations for sentiment metrics.
- Further Exploration of Estimators: Future work could involve exploring a wider range of estimators and their performance characteristics under various data generating processes.
Summary updated