In a standard A/B test, we assume that what happens to user A has no impact on user B. But in "two-sided" marketplaces like Uber, DoorDash, or Lyft, this assumption collapses. If a ride-sharing platform tests a new driver subsidy on only half the drivers in a city, those drivers might stay on the road longer, reducing the number of available trips for the control group. This "interference" makes it impossible to measure the true effect of a policy using traditional randomization.
To solve this, researchers and engineers use switchback experiments. Instead of randomizing across users, they randomize across time. The entire city might receive the treatment for an hour, then switch to the control for the next hour. While this solves the immediate problem of spatial interference, it introduces a new, temporal challenge: the system has a memory.
The Bias-Variance Tug-of-War
The central problem in designing a switchback experiment is deciding how often to flip the switch. As Alex Chin and Sean J. Taylor describe in Bias-Variance Tradeoffs for Designing Simultaneous Temporal Experiments, this choice creates a fundamental trade-off between two types of error.
1. The Variance of "Sticky" Noise
Marketplace data is notoriously noisy. Demand for rides fluctuates based on weather, traffic, and time of day. Crucially, this noise is autocorrelated—if it’s raining now, it’s likely to be raining ten minutes from now.
If we switch treatments very slowly (e.g., every 24 hours), we might accidentally run the treatment during a busy Friday and the control during a quiet Monday. The difference we measure would be mostly noise. To reduce this variance, we want to switch treatments as frequently as possible. Rapidly alternating between conditions allows these environmental trends to "cancel out," providing a more precise estimate of the treatment effect Luo et al. (2022).
2. The Bias of Carryover Effects
However, switching too fast introduces carryover effects. In a marketplace, the state of the system (the number of drivers on the road, their locations, and their current earnings) doesn't reset instantly. A subsidy active in the first hour might influence driver behavior in the second hour, even after the subsidy is turned off.
As Hongtu Zhu notes in An Analysis of Switchback Designs in Reinforcement Learning, frequent switching prevents the system from ever reaching a "steady state." If the switching interval is shorter than the time it takes for the system to settle, the data becomes contaminated by the "ghosts" of previous treatments. This creates a distributional shift penalty, where the measured effect is biased because the system is constantly in a state of transition Wen et al. (2024).
Modeling the System Memory
To find the "Goldilocks" switching frequency, researchers have moved toward modeling the underlying dynamics of the system rather than treating it as a black box.
One approach, ARMA-Design, treats the experiment as a controlled autoregressive moving average process Sun et al. (2024). By explicitly modeling how past states and past noises influence the current outcome, engineers can calculate the exact switching frequency that minimizes the Mean Squared Error (MSE).
Similarly, Shikai Luo and colleagues propose the Varying Coefficient Decision Process (VCDP) to decompose treatment effects into direct and indirect components Luo et al. (2022). This allows for a more granular understanding of how a policy change ripples through a spatio-temporal environment over time.
Practical Design Strategies
When the carryover effects are long-lasting, the optimal strategy is to switch slowly to allow the system to reach equilibrium. When the noise is highly correlated but carryover is short-lived, rapid switching is superior.
Recent work has focused on making these designs more robust:
- Empirical Bayes Designs: Alex Chin and Sean J. Taylor demonstrate that using historical data to estimate the strength of carryover and noise can reduce estimation error by up to 33% Chin & Taylor (2024).
- Minimax Optimality: For scenarios where both spatial spillover and temporal carryover are present, Haoyang Yu and Hanzhong Liu developed a minimax optimal design that minimizes the worst-case variance, providing a rigorous schedule for when to randomize Yu & Liu (2025).
Ultimately, the goal of this research direction is to move beyond "rules of thumb" for A/B testing. By quantifying the interplay between a system's memory (carryover) and its environment's stickiness (autocorrelation), platforms can measure the true impact of their policies in even the most interconnected marketplaces.