The Challenge of Cannibalization in A/B Testing
In complex e-commerce ecosystems, a single A/B test often triggers a "fat hand" effect. For instance, when an engineer at a company like Etsy runs an A/B test on a new recommendation module, they might see clicks on that module skyrocket while sitewide conversion remains flat. This occurs because the new module often "steals" attention from other features, such as organic search. If a user would have purchased an item via search regardless, the recommendation module isn't adding incremental value; it is simply shifting user behavior.
Standard A/B testing measures the Total Effect, which aggregates these behaviors. To truly understand a product's value, we require Causal Mediation Analysis (CMA) to separate the Direct Effect (the intrinsic value added by the feature) from the Indirect Effect (the induced changes in other products). However, traditional CMA relies on the strict "Sequential Ignorability" assumption, which posits that no unmeasured variables affect both the mediator and the outcome—an assumption almost always violated in complex web environments.
A Generalized Framework for Mediation
This paper introduces a robust causal mediation framework designed to handle unmeasured, causally-dependent mediators. The authors define two key metrics:
- Generalized Average Direct Effect (GADE): The effect of the treatment on the outcome that does not pass through the specific mediator of interest.
- Generalized Average Causal Mediation Effect (GACME): The effect of the treatment on the outcome that is mediated through the specific variable being studied.
The authors mathematically prove that under a Linear Structural Equation Model (LSEM), these generalized effects can be identified without measuring every intermediate variable. As long as the treatment ($T$), the outcome ($Y$), and the specific mediator of interest ($M$) are measured, the influence of unmeasured upstream and downstream mediators effectively "collapses" into the coefficients of a simplified model.
Estimation via ITGMM
The proposed method estimates these effects using a system of two linear regressions: one regressing the mediator on the treatment, and another regressing the outcome on the treatment, the mediator, and their interaction. To address potential correlations between error terms—a common consequence of unmeasured mediators—the authors utilize the Iterative General Method of Moments (ITGMM) for consistent and efficient estimation.
Case Study: Disentangling Effects at Etsy
The framework was applied to real-world experimental data from Etsy, revealing insights that standard A/B metrics missed. In one recommendation module test, the total effect (ATE) on conversion was an insignificant +0.22%, despite a +28% lift in clicks.
As shown in the table above, CMA revealed a significant positive direct effect (GADE) of approximately +0.49%, which was being partially offset by a negative indirect effect (GACME) of -0.27% through reduced organic search. This "Hidden Win" demonstrated that the core improvement was real, leading Etsy to launch the feature despite the flat total conversion metric.
Conversely, a test on promoted listings showed significant revenue gains but a slight decrease in sitewide conversion. CMA revealed that the negative indirect effect on organic search was larger in magnitude than the direct gains from ads, indicating that the product was harming overall marketplace health despite appearing successful on a standard dashboard.
Strategic Implications
The authors argue that the Direct Effect (GADE) is often a superior success metric for product teams than the Total Effect, as it represents the intrinsic value of an improvement before cannibalization occurs. By disentangling these effects, organizations can make more informed "launch or kill" decisions, potentially choosing to launch a feature with a strong positive direct effect and then focusing on mitigating its negative indirect impacts later.
Summary updated