Estimating the Value of Evidence-Based Decision Making

Alberto Abadie, Anish Agarwal, Guido Imbens, Siwei Jia, James McQueen, Serguei StepaniantsarXiv 2023 · DOI · arXiv

Abstract

Modern organizations increasingly rely on large-scale experimentation to guide product development and policy decisions. However, traditional statistical frameworks often fail to account for the economic costs and potential payoffs of these decisions. This paper introduces a formal framework to quantify the economic value of Evidence-Based Decision Making (EBDM). By applying empirical Bayes methods to metadata from thousands of historical experiments, we demonstrate how organizations can calculate the return on investment (ROI) for statistical precision. Our findings reveal that conventional "statistical significance" rules are often economically suboptimal, potentially leaving substantial value on the table.

Introduction

In the era of big data, companies like Amazon, Google, and Netflix conduct hundreds of thousands of A/B tests annually. Despite the scale of these operations, a principled approach to valuing the evidence generated by these tests remains elusive. Managers frequently struggle with fundamental resource allocation questions: Is a specific experiment worth its cost? Should sample sizes be increased to improve precision? Are we testing the right ideas?

Current industry standards typically rely on arbitrary statistical conventions, such as requiring 80% power at a 5% significance level. These heuristics ignore the underlying economic reality—the actual costs of data collection and the magnitude of the potential payoffs. We address this gap by proposing the "Value of Evidence" (VoE) metric, treating experimentation as an investment that can be optimized based on expected utility rather than arbitrary p-value thresholds.

Methodology: An Empirical Bayes Framework

The central challenge in valuing evidence is that the true effect ($\tau$) of a proposed change is unobserved; we only possess a noisy estimate ($\hat{\tau}$). To bridge this gap, we employ an Empirical Bayes (EB) framework.

1. Estimating the Prior

We utilize metadata from a large corpus of past experiments to estimate the distribution of "idea quality," which serves as our prior distribution. This allows us to understand the baseline probability of success for new interventions within a specific organizational context.

2. Parametric vs. Nonparametric Estimation

We compare two approaches for modeling the prior:

  • Gaussian Model: Assumes idea quality follows a standard normal distribution.
  • Nonparametric Maximum Likelihood Estimator (NPMLE): A more flexible approach that can capture "fat tails" or "spiky" distributions. This is particularly useful for environments where most ideas have negligible impact, but a small fraction are "home runs."

3. Mitigating the Winner’s Curse

By using the estimated prior to "shrink" noisy experimental results toward the mean, our framework corrects for the "winner’s curse." This phenomenon occurs when the most statistically significant results are those where measurement noise happened to align with the treatment effect, leading to over-optimistic implementation decisions.

4. Counterfactual Analysis

Our model enables researchers to simulate the impact of organizational changes. For instance, we can quantify the total value gained by increasing experimental precision (e.g., through larger sample sizes) or by shifting the focus toward more heterogeneous, high-variance project ideas.

Data: The Upworthy Research Archive

To validate our framework, we analyzed the Upworthy Research Archive, which contains outcomes from 4,857 online experiments. Each experiment tested different headlines and images for the same article to maximize click-through rates. We used the point estimates and standard errors from these thousands of tests as metadata to learn the underlying distribution of treatment effects in the Upworthy ecosystem.

Key Insights

The Suboptimality of P-Values

Our analysis demonstrates that decision rules based strictly on statistical significance ($\alpha = 0.05$) are fundamentally suboptimal for value maximization. Significance testing prioritizes Type I error control (avoiding false positives) over the expected payoff. In business contexts, a "false positive" that still has a positive (though small) expected value is often preferable to discarding a potentially beneficial intervention due to lack of "significance."

Precision vs. Ideation

The value of evidence is intrinsically linked to the variance of idea quality. If a team generates a wide variety of ideas—ranging from very poor to excellent—high-precision evidence becomes incredibly valuable for filtering. Conversely, if all ideas are mediocre and similar, investing in high-precision testing yields diminishing returns.

Key Results

  • Value Gains: A VoE-based decision rule—implementing any intervention with a positive posterior expected value—yielded significantly higher returns than a 5% significance rule.
  • The Cost of Significance: In the Upworthy context, strict significance screening discarded approximately 30% of attainable value.
  • ROI of Precision: Reducing estimation variance by half (an investment in measurement precision) increased the total value of EBDM by 8.33%.
  • ROI of Ideation: Increasing the heterogeneity of ideas by 50% led to a 34.32% gain in value. This suggests that for organizations like Upworthy, investing in better "idea generation" is significantly more impactful than investing in "better measurement."

Conclusion and Limitations

While our framework provides a robust method for valuing evidence, several considerations remain for future research. Currently, the model assumes a risk-neutral decision-maker; risk-averse organizations might still prefer stricter significance thresholds to avoid downside risk. Additionally, the model assumes Gaussian measurement noise, which may not hold for highly skewed outcomes. Finally, we treat the distribution of ideas as fixed, though future work could explore how value-based decision rules might incentivize employees to propose more ambitious "moonshot" projects.

By shifting the focus from p-values to expected utility, organizations can move beyond statistical conventions and make more economically sound decisions about their experimental pipelines.

Summary updated

Abstract

In an era of data abundance, statistical evidence is increasingly critical for business and policy decisions. Yet, organizations lack empirical tools to assess the value of evidence-based decision making (EBDM), optimize statistical precision, and balance the costs of evidence-gathering strategies against their benefits. To tackle these challenges, this article introduces an empirical framework to estimate the value of EBDM and evaluate the return on investment in statistical precision and project ideation. The framework leverages parametric and nonparametric empirical Bayes methods to account for parameter heterogeneity and measure how statistical precision changes the value of evidence. The value extracted from statistical evidence depends critically on how organizations translate evidence into policy decisions. Commonly used decision rules based on statistical significance can leave substantial value unrealized and, in some cases, generate negative expected value.
BibTeX
@misc{abadie2023estimating,
  author = {Alberto Abadie and Anish Agarwal and Guido Imbens and Siwei Jia and James McQueen and Serguei Stepaniants},
  title = {{Estimating the Value of Evidence-Based Decision Making}},
  year = {2023},
  eprint = {2306.13681},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2306.13681},
}
Figure