Is the Performance of My Deep Network Too Good to Be True? A Direct Approach to Estimating the Bayes Error in Binary Classification

Takashi Ishida, Ikko Yamane, Nontawat Charoenphakdee, Gang Niu, Masashi Sugiyama

Introduction

In the pursuit of State-of-the-Art (SOTA) performance, researchers often celebrate marginal improvements in error rates. However, every classification task has a fundamental limit: the Bayes error. This irreducible error arises from inherent ambiguities—such as an image that genuinely shares features of two classes or medical data where experts disagree. Knowing this limit is vital; if a model's error rate drops below the Bayes error, it is likely overfitting the test set rather than learning robust patterns.

A Direct, Instance-Free Approach

Traditional methods for estimating Bayes error often require training complex auxiliary models or accessing raw data instances (e.g., the original images). This research introduces a method that is both model-free and instance-free. Following Vapnik’s Principle—"do not solve a more general problem as an intermediate step"—the authors estimate the Bayes error directly from the distribution of labels or confidence scores.

Key technical contributions include:

  1. Direct Estimation: For clean "soft labels" (probabilities $p$), the Bayes error is the average of $\min(p, 1-p)$.
  2. Noise Correction: The authors developed an unbiased estimator that remains accurate even when the input probabilities are noisy or "jittery," provided the basic class likelihood (the "sign") is preserved.
  3. Positive-Confidence (Pconf) Estimation: A mathematical framework to estimate the total Bayes error using only positive samples and their associated confidence scores.
The proposed estimator demonstrates superior convergence and accuracy across various experimental setups (A-D) compared to ensemble and hard-classifier baselines.

Experimental Validation

The study utilized high-profile benchmarks like CIFAR-10 and introduced Fashion-MNIST-H, a new dataset featuring ~67 human annotations per image to provide high-quality uncertainty ground truth.

As shown in the experimental results, the proposed method converges to the true Bayes error significantly faster than traditional ensemble methods or those relying on training multi-layer perceptrons. Furthermore, the noise-corrected version of the estimator maintains high fidelity even in the presence of label jitter.

Validation of the noise-corrected estimator showing its ability to recover the true Bayes error (dashed line) even when provided with noisy labels, compared to standard estimators.

Key Insights: Reaching the Ceiling

The most striking finding is that top-tier models, such as the Vision Transformer (ViT), have essentially reached the Bayes error on datasets like CIFAR-10. On the CIFAR-10H variant (animals vs. artifacts), the estimated Bayes error is ~1.1%, while ViT achieves a test error of ~1.2%. This suggests that for these benchmarks, we have reached the theoretical limit of performance.

The method was also applied to real-world data from ICLR OpenReview. The estimated Bayes error for the peer-review process (accept/reject decisions) was found to be between 8.0% and 9.6%. This quantifies the inherent subjectivity and "noise floor" of scientific reviewing that no automated system is likely to surpass.

Conclusion

By providing a simple, direct way to calculate the Bayes error, this work offers a "sanity check" for the machine learning community. It highlights that in several mature benchmarks, the focus should perhaps shift from chasing 0.1% improvements to addressing data quality or exploring more complex, less-saturated tasks. Future work will look to extend these instance-free estimators to multi-class problems like ImageNet.

Summary updated

BibTeX
@inproceedings{ishida2023is,
  author = {Takashi Ishida and Ikko Yamane and Nontawat Charoenphakdee and Gang Niu and Masashi Sugiyama},
  title = {{Is the Performance of My Deep Network Too Good to Be True? A Direct Approach to Estimating the Bayes Error in Binary Classification}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=FZdJQgy05rz},
}
Figure