TL;DR
Estimating the long-term impact of product changes using A/B tests is crucial but challenging due to the cost and time required for long experiments. This paper introduces a "longitudinal surrogate framework" that uses short-term A/B test data to accurately predict these long-term effects, outperforming existing methods.
What Problem Does This Paper Address?
In the world of online platforms and tech companies, A/B testing is the gold standard for evaluating new features or product updates. However, many product changes are intended to have long-term effects, and ideally, A/B tests should run for months or even years to capture these. In practice, this is often infeasible due to the high costs of occupying user traffic and delaying product iteration. As a result, companies often rely on short-term A/B test results to make decisions about long-term product strategies. This can be misleading because short-term effects can be influenced by novelty or primacy effects, which may not persist over time. The core problem is how to reliably estimate the long-term effects of long-term treatments using only short-term experimental data.
Methods
The paper proposes a longitudinal surrogate framework. The core idea is to use "longitudinal surrogates," which are intermediate outcomes that act as bridges between the treatment and the ultimate long-term outcome.
Key components of the framework include:
- Longitudinal Surrogacy Assumption: This is a crucial assumption stating that the treatment assignment at an earlier period is independent of future outcomes, conditional on intermediate outcomes at a middle period. This is a generalization of the standard surrogacy assumption.
- Longitudinal Surrogate Index: This is a function that predicts future primary outcomes using intermediate outcomes, pre-treatment variables, and treatment assignments. It's time-dependent and differs from previous definitions.
- Pivot Index: This function helps bridge surrogates across different time periods. It's essentially a conditional expectation of future surrogates given past surrogates, pre-treatment variables, and treatment assignments.
- Linear Surrogate Model: To address challenges with large sample sizes and estimation, the authors introduce a linear functional form for the surrogate index and pivot index. This simplifies the estimation process and makes it more practical.
The framework involves iterative estimation, where predictions from one period are used as inputs for the next, allowing extrapolation into the future.
Datasets
The proposed framework is validated using data from two real-world long-term experiments conducted on WeChat Search. These experiments involved over a million users each and provided valuable "ground truth" data for evaluating the model's performance. The paper also utilizes synthetic experiments to explore scenarios not covered by the real-world data and to test the robustness of the method under various assumption violations.
Key Insights and Ideas
- Short-term A/B tests can predict long-term effects: The most significant insight is that it's possible to reliably estimate long-term treatment effects using data from shorter experiments, overcoming a major practical hurdle in A/B testing.
- Surrogates are key, but their longitudinal nature matters: The framework highlights that intermediate outcomes (surrogates) are essential, but their temporal relationships and how they evolve over time are critical for accurate long-term prediction.
- Linearity assumption is surprisingly effective: While non-linear models are often powerful, the paper finds that a linear surrogate model, when combined with a rich set of surrogates, often outperforms non-linear approaches in practice due to lower variance and more reliable confidence intervals.
- The "carryover effect" is not enough: The paper argues that existing methods focusing solely on "carryover effects" (impact of past treatments on future outcomes) are insufficient for long-term treatments, as they miss the "direct effect" of ongoing treatments in the future.
Key Results
- The proposed longitudinal surrogate framework consistently outperforms baseline methods (Constant Extrapolation and Vector Autoregressive Model) in terms of bias and mean squared error (MSE) on real-world WeChat data.
- In Experiment 1, the approach reduced estimation bias by 59.8% on average across experimental periods, without increasing MSE.
- The model successfully captured increasing trends in treatment effects in Experiment 1, a capability that baseline models lacked.
- In Experiment 2, the model effectively captured decreasing trends in treatment effects, outperforming baselines in bias across various experimental period lengths.
- Sensitivity analyses showed the method's robustness to mild violations of assumptions (surrogacy, comparability, linearity), especially with longer experimental periods or richer surrogate sets.
Limitations and Future Work
- Assumption 1 (Longitudinal Surrogacy) is not directly testable: The authors acknowledge this limitation and propose sensitivity analysis to assess its impact.
- Non-linear data-generating processes can be challenging: While the linear model is robust to moderate non-linearities, strongly non-linear scenarios can cause the method to break down.
- Curse of dimensionality: Choosing an appropriate number of surrogates is crucial; too few can lead to bias, and too many can cause overfitting. Techniques like elastic net regularization are suggested to mitigate this.
- Future work: The authors suggest integrating their framework with optimal stopping strategies in A/B testing to determine the ideal experimental period length. They also propose incorporating structural information, like user behavior models, to improve estimation, especially when data is scarce.
Summary updated