Online Testing of Subgroup Treatment Effects Based on Value Difference

Miao Yu, Wenbin Lu, Rui Song

Abstract

Online A/B testing is a cornerstone of product development, yet traditional fixed-horizon frameworks face two major challenges: inflated false-positive rates under continuous monitoring and the inability to identify beneficial subgroups due to the homogeneous treatment effect assumption. We propose SUBTLE (SUBgroup Treatment effects based on vaLuE difference), a sequential testing framework that addresses these issues simultaneously. SUBTLE allows for continuous "peeking" without compromising statistical guarantees and identifies subgroups that benefit from a treatment even when the average population effect is negligible. Our results demonstrate that SUBTLE maintains high detection power, controls Type I error, remains robust to noise covariates, and enables early stopping.

Introduction

In the high-tech industry, A/B testing guides innovation by determining the superior variant between two options. However, standard methodologies often rely on a "one-size-fits-all" assumption, looking only at the Average Treatment Effect (ATE). This can lead to discarding features that significantly benefit specific user segments while having a neutral or negative impact on others. Furthermore, the "peeking problem"—where researchers monitor results before a pre-determined sample size is reached—often leads to inflated Type I errors.

SUBTLE bridges the gap between sequential testing and subgroup analysis. By shifting the focus from ATE to "Value Difference," SUBTLE identifies if any subgroup exists that would benefit from the treatment, providing a more nuanced and powerful tool for personalized product development.

Methodology

The SUBTLE framework is built on several key pillars:

  • Value Difference ($\Delta$): Instead of comparing group means, SUBTLE tests the difference in outcomes between an optimal treatment rule (treating only those who benefit) and a control rule (treating no one). A positive value difference indicates the existence of a beneficial subgroup.
  • Sequential Monitoring via mSPRT: By utilizing a mixture Sequential Probability Ratio Test (mSPRT), SUBTLE constructs a martingale-based test statistic. This ensures that the Type I error remains controlled regardless of how frequently the data is monitored.
  • Double Robustness: The use of Augmented Inverse Probability Weighted (AIPW) estimators ensures that the test remains valid as long as either the outcome model or the treatment assignment model is correctly specified.
  • Machine Learning Integration: SUBTLE employs Random Forests to estimate nuisance parameters in a batch-wise fashion, allowing it to handle complex, non-linear relationships in high-dimensional data.

Experimental Evaluation

The performance of SUBTLE was evaluated using both synthetic simulations and the Yahoo! Today Module dataset.

Robustness to Noise

In real-world scenarios, datasets are often cluttered with irrelevant variables. SUBTLE demonstrates remarkable robustness to these "noise covariates." While competing methods like the Sequential Score Test (SST) see a significant drop in power as noise increases, SUBTLE maintains consistent performance.

Power vs Number of noise covariates for SUBTLE and SST

Power and Early Stopping

SUBTLE consistently outperforms traditional methods in high-dimensional settings. For instance, in complex simulations (Model V), SUBTLE achieved a detection power of 0.997, significantly higher than the 0.742 achieved by SST.

Crucially, as a sequential test, SUBTLE can terminate as soon as a significant effect is detected. This "early stopping" capability reduces opportunity costs by allowing winning features to be deployed sooner.

Distribution of stopping times for the SUBTLE test

Real-World Application: Yahoo! Dataset

On the Yahoo! Today Module dataset, SUBTLE identified a user subgroup with a click-through rate (CTR) improvement of 0.009, compared to a population-wide improvement of only 0.006. This conclusion was reached with a sample size of just 12,400, demonstrating the efficiency of the sequential approach in a live environment.

Conclusion

SUBTLE provides a robust and flexible framework for online experimentation. By moving beyond average effects and enabling continuous monitoring, it allows organizations to identify hidden value in their products more efficiently and accurately. Future work may explore incorporating adaptive allocation strategies to further optimize the user experience during the testing phase.

Summary updated

BibTeX
@inproceedings{yu2021online,
  author = {Miao Yu and Wenbin Lu and Rui Song},
  title = {{Online Testing of Subgroup Treatment Effects Based on Value Difference}},
  booktitle = {ICDM},
  year = {2021},
  doi = {10.1109/icdm51629.2021.00189},
}
Figure