Abstract
Online experimentation, also known as A/B testing, is the gold standard for measuring product impacts and making business decisions in the tech industry. The validity and utility of experiments, however, hinge on unbiasedness and sufficient power. In two-sided online marketplaces, both requirements are called into question. Bernoulli randomized experiments are biased because treatment units interfere with control units through market competition, violating the "stable unit treatment value assumption" (SUTVA). Furthermore, experimental power is often insufficient due to disparate sample sizes on the two sides of the market. This paper proposes a budget-split experimental design that is unbiased in any marketplace where buyers have a defined budget. We demonstrate that it is more powerful than existing unbiased designs and provide a generalizable system architecture for deployment. Finally, we confirm our findings with empirical performance from experiments run in two real-world online marketplaces at LinkedIn.
The Challenge: Interference and Power in Marketplaces
In a standard A/B test, we assume that the treatment of one group does not affect the control group. In marketplaces like LinkedIn Ads or Uber, this assumption is almost always broken. If a new matching algorithm helps a subset of sellers (members) sell more, and the buyers (advertisers) have a fixed budget, those extra sales are often "stolen" from the control group rather than representing true platform growth. This is known as cannibalization bias, and it leads to a significant overestimation of a feature's impact.
Moreover, marketplaces often suffer from a "power" problem. While there may be millions of members, there are often far fewer advertisers. Randomizing on the advertiser side results in small sample sizes and low statistical power, making it difficult to detect meaningful changes. Conversely, randomizing on the member side maximizes power but maximizes bias.
The Budget-split Design
To solve these dual problems, we propose the Budget-split Design. Instead of simple randomization, this design structurally decouples the marketplace into two independent, parallel universes:
- Member Splitting: Users (members) are randomly assigned to two buckets (Bucket 0 and Bucket 1).
- Budget Splitting: Every advertiser’s budget is split exactly in half.
- Parallel Marketplaces: Two independent marketplaces are created. Marketplace 0 uses Bucket 0 users and the first half of the budgets; Marketplace 1 uses Bucket 1 users and the second half.
- Treatment Assignment: One marketplace acts as the control (standard algorithm), while the other receives the treatment.
Because the budgets are physically separated, a "win" in the treatment group cannot cannibalize the budget from the control group. This design essentially creates a "clone" of every advertiser, allowing researchers to observe how the exact same advertiser performs under treatment versus control simultaneously—a significant advancement for causal inference.
Empirical Validation and Results
The methodology was validated using empirical data from two real-world online marketplaces at LinkedIn: the Ads Marketplace and the Hiring Marketplace. The budget-split design was compared against traditional Bernoulli randomization and switchback designs (where the entire site flips between A and B over time).
Key Findings:
- Massive Power Gains: In one marketplace, for an effect size where budget-split had 80% power, traditional campaign-level and switchback experiments had only ~5% power—essentially no better than a coin flip.
- Bias Confirmation: The study confirmed that standard member-level A/B tests resulted in 1x to 2x overestimation of treatment effects due to cannibalization.
- Real-world Impact: This design allowed for the detection of negative impacts on the order of millions of dollars in annual revenue that previously went unnoticed because they were hidden by the noise of lower-power designs.
Implementation and Future Work
Implementing this design requires deep changes to core ad server architecture, specifically the pacing and bidding modules, which must handle "per-treatment" feedback loops. While the design is highly effective for continuous budgets, future research is needed to address marketplaces with "lumpy" or very small discrete budgets (e.g., an advertiser spending only $1). Additionally, refining variance estimation for this complex design remains an area for mathematical improvement.
By splitting the budget—the fundamental source of interference—this design provides a robust framework for trustworthy experimentation in the modern online economy.
Summary updated