Abstract
We present a method for estimating causal effects in time series data when fine-grained information about the outcome of interest is available. Specifically, we examine what we call the split-door setting, where the outcome variable can be split into two parts: one that is potentially affected by the cause being studied and another that is independent of it, with both parts sharing the same (unobserved) confounders. We show that under these conditions, the problem of identification reduces to that of testing for independence among observed variables, and present a method that uses this approach to automatically find subsets of the data that are causally identified. We demonstrate the method by estimating the causal impact of Amazon's recommender system on traffic to product pages, finding thousands of examples within the dataset that satisfy the split-door criterion. Unlike past studies based on natural experiments that were limited to a single product category, our method applies to a large and representative sample of products viewed on the site. In line with previous work, we find that the widely-used click-through rate (CTR) metric overestimates the causal impact of recommender systems; depending on the product category, we estimate that 50-80% of the traffic attributed to recommender systems would have happened even without any recommendations. We conclude with guidelines for using the split-door criterion as well as a discussion of other contexts where the method can be applied.
Introduction
Causal inference is often described as the "holy grail" of data science. Distinguishing between simple correlation and true causation is notoriously difficult due to unobserved confounders—hidden variables that influence both the cause ($X$) and the effect ($Y$). While Randomized Controlled Trials (RCTs) are the gold standard for unbiased estimation, they are often impractical, expensive, or ethically unfeasible in large-scale digital systems.
Traditional alternatives include the back-door criterion, which requires observing all relevant confounders, and instrumental variables, which require finding a variable $Z$ that affects $X$ but not $Y$ directly. This paper introduces the Split-door criterion, a novel framework that allows researchers to automatically discover "natural experiments" within massive datasets by leveraging auxiliary outcomes.
The Split-door Criterion
The core innovation of the Split-door criterion relies on splitting the outcome variable $Y$ into two distinct components:
- $Y_R$ (Referred): The part of the outcome directly reachable through the cause $X$ (e.g., clicking a recommendation link).
- $Y_D$ (Direct): An auxiliary outcome that is independent of $X$ but shares the same unobserved confounders $U_Y$ (e.g., searching for the same product directly).
The identification strategy relies on testing for statistical independence between $X$ and $Y_D$. If $X$ and $Y_D$ are independent in a specific subset of data, it implies that the unobserved confounder $U_Y$ is not acting on $X$ in that instance. Consequently, the relationship between $X$ and $Y_R$ is "unconfounded," allowing the data to be treated as a natural experiment for unbiased causal estimation.
Empirical Application: Amazon Recommendations
The researchers applied this method to a massive dataset of Amazon.com browsing logs, encompassing 23.4 million page visits across 1.3 million unique products. By using URL parameters (ref tags), they distinguished between visits referred by recommendations ($Y_R$) and direct visits via search or navigation ($Y_D$).
The algorithm automatically identified over 7,000 valid natural experiments across diverse product categories. This automated search represents a significant advancement over previous manual methods that relied on rare external shocks (like celebrity mentions).
Key Results and Implications
The study reveals a staggering bias in standard observational metrics. The widely-used Click-Through Rate (CTR) was found to overestimate the causal impact of recommendations by 50% to 80%. While the naive observational CTR was approximately 9.6%, the Split-door estimate revealed the true causal CTR was only about 2.6%.
This suggests that a vast majority of traffic attributed to recommender systems would have occurred even without the recommendations, as users were already likely to navigate to those products through other means.
Conclusion
The Split-door criterion provides a robust, automated way to identify causal effects in high-dimensional time-series data. By utilizing auxiliary outcomes as "canaries" for confounding, researchers can obtain accurate causal insights without the need for explicit RCTs. The authors have released an R package to facilitate the application of this criterion to other domains, such as social media feeds and digital advertising.
Summary updated