Abstract
Modern statistics provides an ever-expanding toolkit for estimating unknown parameters. Consequently, applied statisticians frequently face a difficult decision: retain a parameter estimate from a familiar method or replace it with an estimate from a newer or more complex one. While it is traditional to compare estimates using risk, such comparisons are rarely conclusive in realistic settings. In response, we propose the "c-value" as a measure of confidence that a new estimate achieves smaller loss than an old estimate on a given dataset. We show that it is unlikely that a large c-value coincides with a larger loss for the new estimate. Therefore, just as a small p-value supports rejecting a null hypothesis, a large c-value supports using a new estimate in place of the old. For a wide class of problems and estimates, we show how to compute a c-value by first constructing a data-dependent high-probability lower bound on the difference in loss. The c-value is frequentist in nature, but we show that it can provide validation of shrinkage estimates derived from Bayesian models in real data applications involving hierarchical models and Gaussian processes.
Introduction
In contemporary statistical practice, researchers are often presented with a choice between a standard, well-understood estimator—such as the Maximum Likelihood Estimate (MLE)—and a more sophisticated alternative, such as a hierarchical Bayesian model or a regularized regression. The standard criterion for choosing between these is risk, defined as the expected loss averaged over all possible realizations of the data.
However, risk-based comparisons suffer from two significant limitations. First, they are often inconclusive; one estimator rarely dominates another across the entire parameter space. Second, risk is an "average" metric that may not reflect the performance on the specific dataset at hand. It is entirely possible for an estimator with lower overall risk to perform worse than the default on a majority of actual datasets. This creates a dilemma for practitioners: is it "safe" to switch to a complex model for the specific data currently being analyzed?
The c-value Framework
To address this, we introduce the c-value ($c$ for "confidence"). The c-value provides a frequentist guarantee that a new estimate achieves a smaller loss than the default on the observed dataset. The framework operates by quantifying the "win"—the difference in loss between the default and the alternative—and establishing a high-probability lower bound on this difference.
The methodology involves three primary steps:
- Quantifying the Win: We define the difference in loss between the default estimator and the alternative.
- Constructing a Lower Bound: We derive a data-dependent function, $b(y, \alpha)$, which serves as a high-probability lower bound on the loss difference. This bound must hold uniformly across the entire parameter space.
- Calculating the c-value: The c-value is defined as the largest confidence level $\alpha$ for which the lower bound $b(y, \alpha)$ remains positive.
A high c-value (e.g., $c > 0.95$) indicates strong evidence that the alternative estimator is superior for the specific data observed. This approach mirrors the logic of p-values: just as a small p-value provides confidence to reject a null hypothesis, a large c-value provides confidence to "reject" a default estimator in favor of a more complex one.
Empirical Validation and Applications
The utility of the c-value was tested across several real-world domains where complex models are frequently used to "borrow strength" across groups.
Educational Longitudinal Study
In estimating the average reading ability of students across 676 schools, the c-value was calculated to be 0.9926. This provides overwhelming evidence that a hierarchical Bayesian model, which shrinks individual school means toward a global average, outperforms the school-specific MLEs for this dataset.
Philadelphia Crime Density
The framework was applied to estimate violent crime density across 384 census tracts. When using non-violent crime as auxiliary data to inform the estimates, the c-value exceeded 0.999, justifying the inclusion of the extra data. However, when a further spatial correlation component was added to the model, the c-value dropped to 0.843. This suggests that while the spatial model might have lower risk in theory, there was insufficient evidence in this specific dataset to conclude it was better than the simpler auxiliary-data model.
Ocean Current Dynamics
Using the GLAD dataset, which tracks ocean current velocities via free-floating buoys, we compared a traditional single-scale model to a multi-scale model accounting for mesoscale and submesoscale variations. A c-value of 0.99981 confirmed that the more complex multi-scale model provided significantly more accurate estimates for the observed trajectories.
Discussion and Key Insights
The c-value framework offers several conceptual advantages. Most notably, it provides a frequentist bridge for Bayesian tools. A researcher can use a Bayesian model with a subjective prior and still obtain a frequentist justification for its performance. One does not need to believe the prior is "true" to trust that the resulting estimate is better than the MLE.
Furthermore, the framework explicitly addresses the problem of "double-dipping"—using the same data to both select a model and perform estimation. By constructing bounds that hold uniformly, the c-value maintains rigorous error control even in a two-stage decision process.
While currently derived analytically for specific classes of estimators, future work aims to utilize a parametric bootstrap to create model-agnostic c-values. This would allow the framework to be applied to any pair of estimators, further democratizing the use of complex statistical models with confidence.
Summary updated