Confidently Comparing Estimates with the c-value

Brian L. Trippe, Sameer K. Deshpande, Tamara Broderick

TL;DR

This paper introduces the "c-value," a novel frequentist measure of confidence that a new statistical estimate is better than a familiar one, addressing the common dilemma of choosing between established and complex methods. It provides a principled way to make these decisions based on observed data, offering a more practical alternative to traditional risk-based comparisons.

What Problem Does This Paper Address?

In modern statistics, we're often faced with a plethora of estimation methods. When a new, potentially more sophisticated method emerges, statisticians grapple with the decision of whether to adopt it over a familiar, perhaps simpler, one. Traditional approaches rely on comparing the "risk" of estimators, which is the average loss over all possible datasets. However, this can be misleading: an estimator with lower average risk might still perform worse on the actual dataset observed. This paper tackles the challenge of making confident, data-driven decisions about estimator superiority in realistic scenarios, moving beyond theoretical average performance.

Methods

The core contribution is the c-value, a measure of confidence that a new estimate has a smaller loss than a default estimate on the observed data. The c-value is constructed by first finding a data-dependent, high-probability lower bound on the difference in loss between the two estimates. If this lower bound is positive and large, it provides strong evidence that the new estimate is indeed better. The authors develop theoretical guarantees for the c-value, showing that a large c-value makes it unlikely that the new estimate has a larger loss. They also propose a two-stage estimator that uses the c-value to select between estimates. The paper details how to compute these bounds for various settings, including normal means with affine estimators and correlated noise, and provides extensions to empirical Bayes and logistic regression.

Datasets

The paper uses several real-world datasets for evaluation:

  • Educational Testing Data: Data from the 2002 Educational Longitudinal Study to compare shrinkage estimates with Maximum Likelihood Estimates (MLE).
  • Violent Crime Density Data: Data on violent and non-violent crime counts per square mile in Philadelphia census tracts.
  • Ocean Current Data: Data from the Carthe Grand Lagrangian Drifter (GLAD) deployment dataset to compare Gaussian process models with different kernel choices.

Key Insights and Ideas

  • Data-Driven Confidence: The c-value offers a way to quantify confidence in estimator superiority based on the observed data, which is often more relevant than theoretical average performance (risk).
  • Analogy to p-values: A large c-value serves a similar purpose to a small p-value in hypothesis testing – it provides strong evidence to "reject" the default estimate in favor of a new one.
  • Bridging Bayesian and Frequentist: While the c-value is frequentist in nature, it can validate estimates derived from Bayesian models, offering a way to assess their performance without relying on subjective prior beliefs.
  • Risk vs. Observed Loss: The paper highlights a critical limitation of risk-based comparisons: an estimator with lower risk can still incur higher loss on the majority of observed datasets, a pitfall the c-value aims to mitigate.

Key Results

  • The proposed c-value provides a principled, data-driven framework for comparing statistical estimates.
  • Theoretical guarantees show that a large c-value makes it highly improbable that the alternative estimate has a larger loss than the default.
  • The c-value framework is demonstrated to be effective across various applications, including educational testing, crime rate estimation, and ocean current modeling.
  • In simulations, the c-value approach shows favorable calibration and can detect improvements by alternative estimators, often with greater than nominal coverage. For example, in the educational testing data, a c-value of 0.9926 strongly suggests the empirical Bayes estimate is more accurate than the MLE.

Limitations and Future Work

  • Dimensionality Dependence: The authors suspect c-values might be "underpowered" for robustly identifying improvements in very low-dimensional problems. Further investigation into this dimension dependence is suggested.
  • Model and Estimator Specificity: The construction of the high-probability lower bound is inherently tied to the specific model, loss function, and pair of estimators. Deriving these bounds analytically can be challenging.
  • Generalizing to Broader Models: While extensions to logistic regression are explored, generalizing the c-value framework to a wider range of models, estimates, and loss functions remains an important area for future work.
  • Parametric Bootstrap: The authors propose using a parametric bootstrap as a more model- and loss-agnostic way to compute c-values, especially for low-dimensional problems where the win's distribution depends on a low-dimensional projection of the parameter.

Summary updated

Abstract

Modern statistics provides an ever-expanding toolkit for estimating unknown parameters. Consequently, applied statisticians frequently face a difficult decision: retain a parameter estimate from a familiar method or replace it with an estimate from a newer or more complex one. While it is traditional to compare estimates using risk, such comparisons are rarely conclusive in realistic settings. In response, we propose the “c-value” as a measure of confidence that a new estimate achieves smaller loss than an old estimate on a given dataset. We show that it is unlikely that a large c-value coincides with a larger loss for the new estimate. Therefore, just as a small p-value supports rejecting a null hypothesis, a large c-value supports using a new estimate in place of the old. For a wide class of problems and estimates, we show how to compute a c-value by first constructing a data-dependent high-probability lower bound on the difference in loss. The c-value is frequentist in nature, but we show that it can provide validation of shrinkage estimates derived from Bayesian models in real data applications involving hierarchical models and Gaussian processes. Supplementary materials for this article are available online.
BibTeX
@article{trippe2022confidently,
  author = {Brian L. Trippe and Sameer K. Deshpande and Tamara Broderick},
  title = {{Confidently Comparing Estimates with the c-value}},
  journal = {Journal of the American Statistical Association},
  year = {2022},
  doi = {10.1080/01621459.2022.2153688},
}
Figure