Abstract
Estimating the treatment effect within network structures is a key focus in online controlled experiments, particularly for social media platforms. We investigate a scenario where the unit-level outcome of interest comprises a series of dyadic outcomes, which is pervasive in many social network sources, spanning from microscale point-to-point messaging to macroscale international trades. Dyadic outcomes are of particular interest in online controlled experiments, capturing pairwise interactions as basic units for analysis. The dyadic nature of the data induces interference, as treatment assigned to one unit may affect outcomes involving connected pairs. We propose a novel design-based causal inference framework for dyadic outcomes in randomized experiments, develop estimators of the global average causal effect, and establish their asymptotic properties under different randomization designs. We prove the central limit theorem for the estimators and propose variance estimators to quantify the estimation uncertainty. The advantages of integrating dyadic data in randomized experiments are manifested in a variety of numerical experiments, especially in correcting interference bias. We implement our proposed method in a large-scale experiment on WeChat Channels, assessing the impact of a recommendation algorithm on users' interaction metrics.
Introduction
In standard randomized controlled trials (A/B testing), researchers typically rely on the Stable Unit Treatment Value Assumption (SUTVA), which posits that the treatment assigned to one individual has no impact on the outcome of another. However, in social networks, SUTVA is frequently violated. For instance, if a recommendation algorithm encourages User A to share more content, their friend, User B, will naturally see and interact with more content, even if User B is in the control group.
This phenomenon, known as network interference or "spillover," renders traditional unit-level estimators (such as comparing average clicks between treated and control groups) biased and unreliable. While existing solutions like cluster randomization or exposure mappings exist, they often require precise knowledge of the network structure or restrictive assumptions. This research proposes a more robust framework by shifting the focus to the pairwise nature of interactions—dyadic outcomes.
Methods
The proposed framework disaggregates a user's total activity into specific dyadic interactions, allowing for explicit modeling of interference.
Dyadic Interference Assumption
The core assumption is that the outcome of an interaction between User $i$ and User $j$ (denoted as $Y_{ij}$) depends only on the treatment status of those two specific users, rather than the treatment status of the entire network. This localized interference model is more tractable than global network models.
Estimators and Theory
The authors develop two primary estimators for the Global Average Treatment Effect (GATE):
- Horvitz-Thompson (HT) Type: A weighted average of dyadic outcomes where both users in a pair receive the same treatment (either both treated or both control).
- Hájek Type: A normalized version of the HT estimator that provides greater stability, particularly in scenarios with small sample sizes or extreme treatment probabilities.
Theoretical grounding is established through a Central Limit Theorem (CLT) for these estimators, proving they are asymptotically normal. To address the correlation between dyadic outcomes (e.g., interactions $(i, j)$ and $(i, k)$ both involve user $i$), a conservative variance estimator is proposed using a sensitivity parameter ($\Delta$) to ensure robust uncertainty quantification.
Experimental Results
The methodology was validated through synthetic simulations, real-world network topologies (socfb-Caltech36), and a large-scale empirical application.
Simulation and Bias Correction
In simulations using Barabási–Albert scale-free models, the proposed HT and Hájek estimators exhibited near-zero relative bias. In contrast, traditional unit-level estimators showed biases exceeding 50% in certain network configurations, particularly when treatment probabilities were low (e.g., 5-10%), a common scenario in industry A/B testing.
WeChat Channels Case Study
The framework was applied to a large-scale A/B test on WeChat Channels involving over three million interactions. The goal was to evaluate a recommendation algorithm designed to promote content sharing.
- Traditional Unit-Level Estimator: Failed to find a statistically significant result (p-value 0.567).
- Dyadic Hájek Estimator: Detected a significant 1.124% increase in activity (p-value 0.024).
This result demonstrates that standard methods may fail to detect real product improvements because they cannot account for the noise introduced by network interference.
Conclusion and Future Work
By leveraging the granularity of dyadic data, this framework provides a computationally efficient and theoretically sound method for causal inference in networked environments. While the current model assumes "two-hop" interference (limited to the pair), future work may extend this to "three-hop" or higher-order interference through sensitivity analysis. Additionally, extending this design-based approach to observational data remains a promising avenue for research, potentially integrating propensity score modeling into the dyadic framework.
Summary updated