Trustworthy Online Marketplace Experimentation with Budget-split Design

Min Liu, Jialiang Mao, Kang KangarXiv 2020 · DOI · arXiv

Abstract

Online experimentation, also known as A/B testing, is the gold standard for measuring product impacts and making business decisions in the tech industry. The validity and utility of experiments, however, hinge on unbiasedness and sufficient power. In two-sided online marketplaces, both requirements are called into question. Bernoulli randomized experiments are biased because treatment units interfere with control units through market competition, violating the "stable unit treatment value assumption" (SUTVA). Furthermore, experimental power is often insufficient due to disparate sample sizes on the two sides of the market. This paper proposes a budget-split experimental design that is unbiased in any marketplace where buyers have a defined budget. We demonstrate that it is more powerful than existing unbiased designs and provide a generalizable system architecture for deployment. Finally, we confirm our findings with empirical performance from experiments run in two real-world online marketplaces at LinkedIn.

The Challenge: Interference and Power in Marketplaces

In a standard A/B test, we assume that the treatment of one group does not affect the control group. In marketplaces like LinkedIn Ads or Uber, this assumption is almost always broken. If a new matching algorithm helps a subset of sellers (members) sell more, and the buyers (advertisers) have a fixed budget, those extra sales are often "stolen" from the control group rather than representing true platform growth. This is known as cannibalization bias, and it leads to a significant overestimation of a feature's impact.

Moreover, marketplaces often suffer from a "power" problem. While there may be millions of members, there are often far fewer advertisers. Randomizing on the advertiser side results in small sample sizes and low statistical power, making it difficult to detect meaningful changes. Conversely, randomizing on the member side maximizes power but maximizes bias.

The Budget-split Design

To solve these dual problems, we propose the Budget-split Design. Instead of simple randomization, this design structurally decouples the marketplace into two independent, parallel universes:

  1. Member Splitting: Users (members) are randomly assigned to two buckets (Bucket 0 and Bucket 1).
  2. Budget Splitting: Every advertiser’s budget is split exactly in half.
  3. Parallel Marketplaces: Two independent marketplaces are created. Marketplace 0 uses Bucket 0 users and the first half of the budgets; Marketplace 1 uses Bucket 1 users and the second half.
  4. Treatment Assignment: One marketplace acts as the control (standard algorithm), while the other receives the treatment.
Illustration of the mapping between advertisers, campaigns, and member requests in a marketplace environment.

Because the budgets are physically separated, a "win" in the treatment group cannot cannibalize the budget from the control group. This design essentially creates a "clone" of every advertiser, allowing researchers to observe how the exact same advertiser performs under treatment versus control simultaneously—a significant advancement for causal inference.

Empirical Validation and Results

The methodology was validated using empirical data from two real-world online marketplaces at LinkedIn: the Ads Marketplace and the Hiring Marketplace. The budget-split design was compared against traditional Bernoulli randomization and switchback designs (where the entire site flips between A and B over time).

Key Findings:

  • Massive Power Gains: In one marketplace, for an effect size where budget-split had 80% power, traditional campaign-level and switchback experiments had only ~5% power—essentially no better than a coin flip.
  • Bias Confirmation: The study confirmed that standard member-level A/B tests resulted in 1x to 2x overestimation of treatment effects due to cannibalization.
  • Real-world Impact: This design allowed for the detection of negative impacts on the order of millions of dollars in annual revenue that previously went unnoticed because they were hidden by the noise of lower-power designs.

Implementation and Future Work

Implementing this design requires deep changes to core ad server architecture, specifically the pacing and bidding modules, which must handle "per-treatment" feedback loops. While the design is highly effective for continuous budgets, future research is needed to address marketplaces with "lumpy" or very small discrete budgets (e.g., an advertiser spending only $1). Additionally, refining variance estimation for this complex design remains an area for mathematical improvement.

By splitting the budget—the fundamental source of interference—this design provides a robust framework for trustworthy experimentation in the modern online economy.

Summary updated

Abstract

Online experimentation, also known as A/B testing, is the gold standard for measuring product impacts and making business decisions in the tech industry. The validity and utility of experiments, however, hinge on unbiasedness and sufficient power. In two-sided online marketplaces, both requirements are called into question. The Bernoulli randomized experiments are biased because treatment units interfere with control units through market competition and violate the "stable unit treatment value assumption"(SUTVA). The experimental power on at least one side of the market is often insufficient because of disparate sample sizes on the two sides. Despite the important of online marketplaces to the online economy and the crucial role experimentation plays in product improvement, there lacks an effective and practical solution to the bias and low power problems in marketplace experimentation. Our paper fills this gap by proposing an experimental design that is unbiased in any marketplace where buyers have a defined budget, which could be finite or infinite. We show that it is more powerful than all other unbiased designs in literature. We then provide generalizable system architecture for deploying this design to online marketplaces. Finally, we confirm our findings with empirical performance from experiments run in two real-world online marketplaces.
BibTeX
@misc{liu2020trustworthy,
  author = {Min Liu and Jialiang Mao and Kang Kang},
  title = {{Trustworthy Online Marketplace Experimentation with Budget-split Design}},
  year = {2020},
  eprint = {2012.08724},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2012.08724},
}
Figure