TL;DR
This paper bridges the gap between frequentist and Bayesian hypothesis testing in the context of A/B testing for binomial outcomes. It establishes a direct link between the Bayesian Bayes factor and the frequentist Welch's t-statistic, showing that Bayesian sample size requirements for detecting differences are surprisingly similar to frequentist ones. This work offers a more unified understanding of statistical inference for A/B testing.
What Problem Does This Paper Address?
A/B testing is a ubiquitous method for comparing two versions of something (e.g., a webpage, an ad) to see which performs better, often measured by conversion rates. Traditionally, this has been approached using frequentist methods like the t-test, which focus on p-values. However, Bayesian methods, using Bayes factors, offer an alternative framework that quantifies evidence for one hypothesis over another. A key challenge has been understanding how these two approaches relate, particularly in terms of sample size requirements. Specifically, this paper aims to:
- Quantify the relationship between Bayes factors and frequentist statistics in the common scenario of binomial A/B testing.
- Derive bounds on Bayes factors that connect them to well-known frequentist measures.
- Provide practical insights into sample size requirements for Bayesian A/B testing, comparing them to frequentist rules of thumb.
Methods
The core of the paper lies in its theoretical derivations connecting information-theoretic measures with statistical tests. The authors leverage:
- Bayes Factors: Defined as the ratio of the probability of the data under the null hypothesis ($H_0$) to the probability of the data under the alternative hypothesis ($H_a$).
- Jensen-Shannon Divergence (JS divergence): A symmetric measure of similarity between two probability distributions, derived from the Kullback-Leibler (KL) divergence. The paper shows that the Bayes factor for binomial A/B testing is controlled by the JS divergence between the observed success rates of the two groups.
- Welch's t-statistic: A standard frequentist test for comparing the means of two independent samples with potentially unequal variances. The paper demonstrates that the JS divergence can be bounded by the Welch's t-statistic.
- Information Geometry: The paper elegantly applies concepts from information geometry, particularly the relationship between divergences and statistical distances, to derive these bounds.
The derivation involves showing that the Bayes factor exponent is related to the KL divergence between observed proportions and hypothesized means, and then using inequalities (like Pinsker's inequality) to connect these to the JS divergence and subsequently to the Welch's t-statistic.
Datasets
Not discussed in the paper. The paper focuses on theoretical derivations and establishes general bounds applicable to any binomial A/B testing scenario with observed data.
Key Insights and Ideas
- Bayes Factor as a Measure of Divergence: The paper reveals that the strength of evidence provided by a Bayes factor in binomial A/B testing is directly proportional to the Jensen-Shannon divergence between the success rates of the two groups. This provides an intuitive interpretation: larger differences in observed rates lead to stronger evidence against the null hypothesis.
- Bridging Bayesian and Frequentist Worlds: The most significant insight is the direct mathematical link established between the Bayes factor and the Welch's t-statistic. This demonstrates that, under certain conditions and for specific hypothesis formulations, the Bayesian evidence against a "zero-effect" hypothesis is fundamentally related to the frequentist measure of difference.
- Unified Sample Size Bounds: By connecting the Bayes factor to the Welch's t-statistic, the paper shows that the sample size required for Bayesian A/B testing to detect a non-zero effect is remarkably similar to the sample size required for frequentist tests. This challenges the notion that Bayesian methods always require vastly different sample sizes.
- Information Geometry as a Unifying Framework: The paper highlights the power of information geometry tools in providing elegant derivations and a deeper understanding of the relationship between different statistical inference paradigms.
Key Results
The paper's main theoretical results are encapsulated in Theorem 1 and its corollaries:
- Theorem 1: States that the maximum minimum Bayes factor (over null hypotheses and alternative hypotheses) is bounded by $e^{-2r \cdot JS(\hat{p}_1, \hat{p}_2)}$, where $r$ is the number of trials per group and $JS(\hat{p}_1, \hat{p}_2)$ is the Jensen-Shannon divergence between the observed success rates. It further shows this is bounded by $e^{-t_{Welch}^2 / (2r)}$, where $t_{Welch}$ is the Welch's t-statistic.
- Corollary 1 (Bayesian Sample Bound): Derives a sample size rule for Bayesian A/B testing: $r \gtrsim \frac{1}{2JS(\hat{p}_1, \hat{p}_2)}$. This is shown to be comparable to the frequentist rule $r \gtrsim \frac{2(\hat{p}_1(1-\hat{p}_1)+\hat{p}_2(1-\hat{p}_2)}{(\hat{p}_1-\hat{p}_2)^2}$.
- Comparison: Figure 1 visually compares these bounds, illustrating their similarity for a 10% relative uplift. The paper notes that Bayesian approaches can be more conservative, requiring stronger evidence to reject the null hypothesis compared to standard frequentist thresholds.
Limitations and Future Work
- Assumptions on Priors: The derived bounds and sample size rules depend on specific choices of null and alternative hypotheses, particularly the "maximally favorable" ones. In practice, the choice of priors can influence the Bayes factor. The paper acknowledges that more informative priors are often used in practice.
- Specific Hypothesis Formulation: The results are derived for a specific formulation of the null hypothesis ($H_0: p_1 = p_2$) and alternative hypothesis ($H_a: p_1 \neq p_2$). Extending these bounds to more complex hypothesis structures might be challenging.
- Future Work: The authors suggest that this work provides a foundation for understanding data limitations in Bayesian inference and relating them to frequentist rules. Future work could explore these connections further for other statistical models beyond the binomial case. The paper also implicitly suggests that the Bayesian approach, by its nature, can be more conservative than frequentist tests, a point shared with other research.
Summary updated