Abstract
We propose a new framework for online testing of heterogeneous treatment effects (HTE). The proposed test, named the Sequential Score Test (SST), is designed to control Type I error under continuous monitoring while detecting multi-dimensional HTE. By providing an online p-value calculation, SST facilitates convenient real-time monitoring and extends to multiple testing settings by controlling the false discovery rate. Empirical evaluations using simulations and real-world data demonstrate that SST maintains Type I error control, offers higher detection power, and enables faster inference compared to state-of-the-art methods like mSPRT.
Introduction
Standard A/B testing often relies on fixed-horizon Null Hypothesis Statistical Testing (NHST). However, in fast-paced industry environments, stakeholders frequently "peek" at results as data streams in. This practice violates the assumptions of NHST, leading to inflated false positive rates. Furthermore, most tests focus on the Average Treatment Effect (ATE), potentially overlooking significant Heterogeneous Treatment Effects (HTE) where specific subgroups respond differently to a treatment.
The Challenges of Modern A/B Testing
Two primary issues plague traditional online experimentation:
- The Peeking Problem: Stopping an experiment as soon as a p-value crosses a significance threshold significantly increases Type I errors.
- The "Average" Trap: A feature might be highly beneficial for one segment but neutral for another. An ATE-focused test may fail to detect this signal, leading to missed optimization opportunities.
While existing sequential tests like the mixture Sequential Probability Ratio Test (mSPRT) address the peeking problem, they often struggle with HTE and nuisance parameters (e.g., baseline user differences), becoming computationally expensive for large-scale data.
The Sequential Score Test (SST) Framework
The authors introduce the Sequential Score Test (SST) to overcome these limitations. The core innovation is the use of asymptotic score statistics rather than likelihood ratios.
Key Technical Innovations
- Score-Based Approach: SST utilizes the ratio of asymptotic score statistic distributions, allowing for a closed-form integration. This makes it significantly faster to compute in streaming environments than methods requiring numerical integration.
- Nuisance Parameter Handling: SST accounts for baseline individual effects by utilizing maximum likelihood estimators from the control group.
- Always-Valid P-Values: The framework generates a sequence of p-values that remain valid at any stopping time, permitting continuous monitoring without compromising statistical integrity.
- Multiple Testing Extension: SST can be adapted for multiple comparisons (e.g., testing multiple variations) using sequential versions of Bonferroni and Benjamini-Hochberg procedures.
Experimental Evaluation
The performance of SST was evaluated against mSPRT across various Generalized Linear Models (GLMs), including Linear, Logistic, and Log regression.
As shown in Figure 1, SST consistently demonstrates higher detection power. For instance, in logistic regression with normal covariates, SST achieved a power of 0.730, whereas mSPRT only reached 0.356.
Crucially, Figure 2 illustrates that SST maintains stable Type I error control across all models. In contrast, mSPRT shows significantly inflated Type I error rates in certain linear regression scenarios (e.g., reaching 0.201), making it less reliable for those applications.
Real-World Application: Yahoo! News Data
The framework was also tested on a Yahoo! dataset of user click events. The goal was to identify which user features drove engagement across different news articles.
- Efficiency: SST reached statistically significant conclusions after only 19,600 events, compared to 67,600 events required by mSPRT—a 3.5x improvement in speed.
- Sensitivity: In a multiple testing scenario with 45 pairwise comparisons, SST identified 43 significant pairs, while mSPRT found only 23.
Limitations and Future Work
Despite its advantages, SST has areas for further development:
- Stopping Rules: The current framework allows for rejection but does not define a clear point to "accept the null," potentially leading to indefinite waiting if no effect exists.
- Truncation: Real-world experiments have finite durations. Future research will explore how to set maximum sample sizes (truncation) while balancing power and false negative risks.
- Small-Sample Behavior: As SST relies on large-sample asymptotic theory, its performance in very small "micro-experiments" requires further investigation.
Conclusion
The Sequential Score Test provides a robust, computationally efficient framework for modern online experimentation. By enabling continuous monitoring and the detection of heterogeneous effects, it allows data scientists to uncover deeper insights faster while maintaining rigorous statistical control.
Summary updated