Abstract
Two-sided marketplace platforms often run experiments to test the effect of an intervention before launching it platform-wide. A typical approach is to randomize individuals into the treatment group, which receives the intervention, and the control group, which does not. The platform then compares the performance in the two groups to estimate the effect if the intervention were launched to everyone. We focus on two common experiment types, where the platform randomizes individuals either on the supply side or on the demand side. The resulting estimates of the treatment effect in these experiments are typically biased: because individuals in the market compete with each other, individuals in the treatment group affect those in the control group and vice versa, creating interference. We develop a simple tractable market model to study bias and variance in these experiments with interference. We focus on two choices available to the platform: (1) Which side of the platform should it randomize on (supply or demand)? (2) What proportion of individuals should be allocated to treatment? We find that both choices affect the bias and variance of the resulting estimators but in different ways. The bias-optimal choice of experiment type depends on the relative amounts of supply and demand in the market, and we discuss how a platform can use market data to select the experiment type. Importantly, we find in many circumstances, choosing the bias-optimal experiment type has little effect on variance. On the other hand, the choice of treatment proportion can induce a bias-variance tradeoff, where the bias-minimizing proportion increases variance. We discuss how a platform can navigate this tradeoff and best choose the treatment proportion, using a combination of modeling as well as contextual knowledge about the market, the risk of the intervention, and reasonable effect sizes of the intervention.
Introduction
In standard A/B testing, researchers rely on the "Stable Unit Treatment Value Assumption" (SUTVA), which posits that the treatment of one individual does not affect the outcomes of others. In two-sided marketplaces—such as Airbnb, Uber, or Upwork—this assumption is frequently violated. When a customer in a treatment group receives a discount and books a specific listing, that listing is no longer available for customers in the control group. This phenomenon, known as interference, creates significant bias in experimental results. If left unaddressed, platforms may observe a "lift" in their metrics that would not actually materialize if the feature were rolled out to the entire population.
While platforms often resort to complex experimental designs like switchback tests (alternating treatment over time) or cluster randomization (geographical splits), these methods are often engineering-intensive and prone to high variance. This paper investigates whether simpler designs—Customer-Side Randomization (CR) or Listing-Side Randomization (LR)—can be optimized to provide reliable estimates of the Global Treatment Effect (GTE).
Theoretical Framework: The One-Shot Market Model
To analyze the impact of interference, the authors develop a "one-shot" market model that captures the essential mechanics of a transaction in three steps:
- Consideration: Customers identify a set of listings they are interested in.
- Application: Customers apply to a single listing from their consideration set.
- Acceptance: Listings that receive multiple applications select one customer at random to fulfill the transaction.
By analyzing this model in a "large market regime" (where the number of customers and listings scales toward infinity), the authors derive closed-form expressions for the bias and variance of GTE estimators. This mathematical foundation allows for a rigorous comparison of how market conditions—specifically the ratio of demand to supply (denoted as $\lambda$)—influence the accuracy of different experimental designs.
Choosing the Right Side: The Market Balance Rule
A primary contribution of this research is the "Market Balance Rule," which suggests that the optimal side for randomization depends on which side of the market is more constrained:
- Demand-Constrained Markets: In markets with high supply and relatively few customers (low $\lambda$), Customer-Side Randomization (CR) is nearly unbiased. In these scenarios, customers rarely compete for the same listing, so the interference between treatment and control groups is minimal.
- Supply-Constrained Markets: In markets with high demand and limited listings (high $\lambda$), Listing-Side Randomization (LR) is the superior choice. Here, the competition occurs among customers for a single listing; by randomizing at the listing level, the platform can better isolate the treatment effect.
Interestingly, the authors find that choosing the experiment type that minimizes bias usually does not come with a variance penalty. This "free lunch" implies that platforms can significantly improve the reliability of their experiments simply by selecting the randomization side that aligns with their market's current balance.
The Allocation Tradeoff
While choosing the side of randomization is often a win-win for bias and variance, choosing the proportion of users in treatment (the allocation) involves a sharper tradeoff.
Typically, a 50/50 split is the gold standard for minimizing variance (noise). However, the authors demonstrate that more extreme allocations (e.g., 90% treatment) can actually reduce bias in many scenarios. This finding has significant implications for "ramp-up" launches, where a platform gradually increases the treatment group from 1% to 100%. The research suggests that during these early stages, the platform may actually be more accurate at detecting if a feature is harmful (negative GTE) because the bias is lower at the extremes of the allocation spectrum.
Key Results and Theorems
- Theorem 1 (Market Extremes): The authors mathematically prove that the CR estimator becomes unbiased as relative demand $\lambda \to 0$, while the LR estimator becomes unbiased as $\lambda \to \infty$.
- Bias Monotonicity: For "multiplicative" interventions—those that increase interest across the board—the bias changes monotonically with the treatment fraction.
- MSE Optimization: Numeric evaluations show that while a 50/50 split is a robust rule of thumb for minimizing Mean Squared Error (MSE) in smaller markets, the importance of choosing a "bias-optimal" allocation increases as the market scales.
Conclusion and Future Directions
This research provides a practical roadmap for marketplace platforms to navigate the complexities of interference. By understanding the relationship between market balance and randomization side, platforms can make more informed decisions without necessarily resorting to high-variance experimental designs.
However, the authors note that the "one-shot" model is a simplification of reality. Real-world marketplaces are dynamic; a customer who fails to book today may return tomorrow. Future work could extend this framework to dynamic models or explore "two-sided" randomization designs that attempt to neutralize interference by randomizing both customers and listings simultaneously. Additionally, incorporating "screening" mechanisms—where sellers choose buyers based on quality rather than at random—could further refine the accuracy of these estimators in professional service marketplaces.
Summary updated