Overlapping experiment infrastructure: more, better, faster experimentation

Diane Tang, Ashish Agarwal, Deirdre O'Brien, Mike MeyerKDD 2010 · DOI

TL;DR

This paper introduces Google's "overlapping experiment infrastructure," a novel system designed to dramatically increase the number, quality, and speed of live A/B tests. By allowing multiple experiments to run concurrently on overlapping traffic segments, it overcomes the scalability limitations of traditional single-experiment setups, enabling more data-driven decision-making for product development.

What Problem Does This Paper Address?

At Google, making data-driven decisions is paramount, and live traffic experiments (A/B tests) are the primary method for gathering empirical data to evaluate changes. However, as the pace of innovation accelerates, the sheer volume of potential changes and the need to test them quickly creates a significant bottleneck. The challenge is to run more experiments, ensure they lead to better decisions, and complete them faster. Traditional approaches, where each user request can only be part of a single experiment, simply don't scale to the thousands of parameters and potential changes Google wants to test. This paper addresses the need for a more scalable and efficient experimentation infrastructure.

Methods

The core innovation is the overlapping experiment infrastructure, which moves beyond the "single layer" approach where each query is in at most one experiment. Instead, they propose a system with multiple layers, where each layer is associated with a subset of parameters that cannot be varied independently. This allows a single query to participate in multiple experiments simultaneously, one from each layer.

Key concepts include:

  • Domains: Segments of traffic.
  • Layers: Correspond to subsets of system parameters.
  • Experiments: Segments of traffic where zero or more system parameters are given alternate values.

This system allows for nesting of domains and layers, providing flexibility. For instance, a "non-overlapping domain" can exist alongside an "overlapping domain" with multiple layers. They also introduce launch layers for gradual rollouts of new features, which are distinct from regular experiment layers.

To manage this complexity, they emphasize the importance of:

  • Data file checks: Automated validation of experiment configurations.
  • Real-time monitoring: Quick alerts for unexpected metric behavior.
  • Experiment design & sizing tools: Ensuring experiments have enough statistical power.
  • Triggering, logging, & counter-factuals: Accurately measuring impact even when experiments don't affect all diverted traffic.
  • Pre- & post-periods: Validating traffic comparability and detecting learned effects.
  • Fast analytics: Tools for rapid and accurate metric evaluation.
  • Education: Processes like "Experiment Council" and "Interpreting the Data" forums to improve experiment design and analysis.

Datasets

Not discussed in the paper. The paper focuses on the infrastructure and processes for running experiments on live Google search traffic, rather than specific datasets for training models.

Key Insights and Ideas

  • Scalability through Overlap: The fundamental insight is that by allowing experiments to overlap, you can dramatically increase the number of experiments that can run concurrently, directly addressing the scalability bottleneck.
  • Parameter Partitioning is Key: The success hinges on intelligently partitioning system parameters into layers, ensuring that parameters within a layer are interdependent while parameters across different layers can be varied independently.
  • Flexibility via Nesting: The ability to nest domains and layers provides a powerful mechanism to handle diverse experimentation needs, from broad changes to highly specific ones.
  • Infrastructure is Not Enough: The paper strongly emphasizes that a robust infrastructure is only one piece of the puzzle. Tools, educational processes, and a culture of data-driven decision-making are equally critical for successful experimentation.
  • The Importance of Counter-factuals: For experiments that don't affect all diverted traffic, logging counter-factual data in the control is crucial for accurate impact measurement.
  • Standardization Drives Efficiency: Canonical tools for data checks, sizing, and analytics reduce errors, ensure comparability, and speed up the entire experimentation lifecycle.

Key Results

The paper presents trends showing significant improvements after the deployment of the overlapping experiment infrastructure:

  • More Experiments: Nearly an order of magnitude increase in the number of experiments run.
  • More Experimenters: A substantial increase in the number of people running experiments.
  • More Launches: A corresponding increase in the number of features launched.
  • Better Experiments: Anecdotal evidence suggests fewer misconfigured experiments, fewer forgotten experiments, and more consistent metric definitions and calculations due to standardized tools.
  • Faster Experimentation: While implementation of new features remains the slowest phase, the process of pushing experiments, getting initial insights, and analyzing them has become significantly faster.

Limitations and Future Work

Limitations:

  • Anecdotal Data: While trends are presented, much of the "better" and "faster" assessment is based on anecdotal evidence rather than hard quantitative metrics.
  • Complexity of Nesting: While flexible, the nesting of domains and layers can increase complexity.
  • Rare Language Experiments: Carving out sufficient space for statistically significant results for experiments on very small traffic slices (e.g., rare languages) remains a challenge.

Future Work:

  • Expediting Feature Implementation: Developing tools to speed up the initial implementation and prototyping of new features.
  • Radically Different Experiments: Supporting experiments that go beyond parameter changes.
  • Parameter Composability: Enabling more flexible parameter usage across layers through operators like multiplication and addition.
  • Efficient Space Utilization: Developing more expressive conditions for traffic segmentation and robust verification mechanisms.
  • Improving Experiment Space Utilization: Finding ways to run experiments on small traffic slices more efficiently.

Summary updated

BibTeX
@inproceedings{tang2010overlapping,
  author = {Diane Tang and Ashish Agarwal and Deirdre O'Brien and Mike Meyer},
  title = {{Overlapping experiment infrastructure: more, better, faster experimentation}},
  booktitle = {ACM SIGKDD Conference on Knowledge Discovery and Data Mining},
  year = {2010},
  doi = {10.1145/1835804.1835810},
}
Figure