Abstract
Estimating the total treatment effect (TTE) of a new feature in social platforms is crucial for understanding its impact on user behavior. However, the presence of network interference, which arises from user interactions, often complicates this estimation process. Experimenters typically face challenges in fully capturing the intricate structure of this interference, leading to less reliable estimates. To address this issue, we propose a novel approach that leverages surrogate networks and the pseudo inverse estimator. Our contributions can be summarized as follows: (1) We introduce the surrogate network framework, which simulates the practical situation where experimenters build an approximation of the true interference network using observable data. (2) We investigate the performance of the pseudo inverse estimator within this framework, revealing a bias-variance trade-off introduced by the surrogate network. We demonstrate a tighter asymptotic variance bound compared to previous studies and propose an enhanced variance estimator outperforming the original estimator. (3) We apply the pseudo inverse estimator to a real experiment involving over 50 million users, demonstrating its effectiveness in detecting network interference when combined with the difference-in-means estimator. Our research aims to bridge the gap between theoretical literature and practical literature, providing a solution for estimating TTE in the presence of network interference and unknown interference structures.
Introduction
In standard A/B testing, causal inference relies heavily on the Stable Unit Treatment Value Assumption (SUTVA), which posits that the treatment assigned to one individual does not affect the outcomes of others. In the context of social platforms like WeChat, Facebook, or LinkedIn, this assumption is frequently violated. When a user receives a new feature—such as an enhanced video-sharing interface—their increased activity naturally spills over to their connections, influencing the behavior of users in the control group. This phenomenon, known as network interference, biases traditional estimators like the Difference-in-Means (DIM).
While theoretical models for estimating the Total Treatment Effect (TTE) under interference exist, they often assume perfect knowledge of the underlying interaction network. In practice, the "true" network is often unobservable, dynamic, or computationally prohibitive to map. This paper addresses the gap between theoretical ideals and industrial reality by introducing a framework based on approximate interference networks.
Methodology
The Surrogate Network Framework
We propose the use of a surrogate network ($G$) as a proxy for the true, latent interference network ($A$). Instead of attempting to map every possible interaction, experimenters construct $G$ using observable historical data, such as user interactions over a recent 28-day window. This approach acknowledges that while our map of user interference is incomplete, a sparse and well-constructed approximation can still provide significant inferential power.
The Pseudo Inverse (SNIPE) Estimator
To estimate TTE within this framework, we utilize the pseudo inverse estimator, also referred to as SNIPE. This estimator reweights user outcomes based on the treatment assignments of their immediate neighbors within the surrogate network. A critical component of our methodology is the focus on "low-order" interactions (setting the parameter $eta=1$). This simplification prevents variance from exploding in massive networks and maintains computational feasibility for platforms with millions of nodes.
Theoretical Contributions
Tighter Variance Bounds
Previous literature suggested that the variance of network estimators scales cubically with the network degree ($d^3$). Our analysis provides a tighter asymptotic variance bound of $O(d_G^2/np)$, where $d_G$ is the average degree of the surrogate network, $n$ is the number of users, and $p$ is the treatment probability. This theoretical refinement suggests that the estimator is more stable in large-scale applications than previously thought.
The Bias-Variance Trade-off
The choice of the surrogate network $G$ introduces a fundamental trade-off. A denser surrogate network (including more edges) may capture more real-world interference, thereby reducing bias. However, this increased density also increases the variance of the estimate, making the results noisier. Our framework allows experimenters to navigate this trade-off by selecting surrogate structures that balance these two forces.
Empirical Evaluation
Synthetic and Real-World Subsets
We validated our theoretical bounds using Erdos-Rényi synthetic networks, confirming that empirical variance scales with the square of the average degree ($d^2$). Further testing on a real-world social network subset (100,015 nodes and 2.2 million edges) demonstrated that the pseudo inverse estimator consistently outperformed cluster-based randomization, which often fails in social graphs that lack clear, isolated partitions.
Large-Scale Deployment: WeChat
The methodology was deployed on a massive scale within the WeChat ecosystem, involving 53,603,004 users and over 1 billion edges. By comparing the SNIPE estimator with the standard DIM estimator, we developed a diagnostic tool for interference detection. If the two estimators diverge significantly, it provides empirical evidence that SUTVA has been violated. In the WeChat experiment, this approach successfully identified significant treatment effects at a 95% confidence level that were completely obscured by interference when using standard A/B testing metrics.
Discussion and Future Work
Our research demonstrates that the pseudo inverse estimator, supported by a surrogate network framework, is a robust tool for industrial-scale causal inference. However, challenges remain. The construction of the "optimal" surrogate network still requires domain expertise, and extreme unit-level heterogeneity—where users respond to treatments in highly idiosyncratic ways—can still introduce bias into variance estimation. Future research will focus on automating surrogate network construction and exploring higher-order interactions without sacrificing statistical power.
Summary updated