A Simple Way to Deal with Cherry-picking

Junpei Komiyama, Takanori MaeharaarXiv 2018 · arXiv

Abstract

This paper addresses the "reproducibility crisis" in machine learning research caused by "cherry-picking"—the strategic reporting of results only on a subset of datasets where an algorithm performs exceptionally well. We formalize this selection bias and demonstrate how the increasing abundance of public datasets paradoxically makes it easier to falsely claim algorithmic superiority. We propose a "post-reporting" inspector model that utilizes a small set of fresh, independent datasets to detect selection bias, ensuring scientific integrity without sacrificing the statistical power needed to identify genuine innovations.

Introduction

In the competitive landscape of machine learning, "state-of-the-art" (SOTA) results are the primary benchmark for success. However, there is growing concern that many reported improvements are not reproducible. Researchers may only report results on a subset of datasets ($N_P$) where their algorithm outperformed existing ones, a practice known as cherry-picking. Standard statistical tools, such as the $p$-value, fail in this context because they assume datasets are chosen randomly, not strategically to maximize perceived performance.

The Paradox of Data Abundance

A key insight of this research is the "Paradox of Abundance": as the number of available public datasets ($N_A$) grows, it becomes easier for a researcher to find a small subset ($N_P$) where even a mediocre algorithm appears significantly better than the baseline. When $N_A$ is large, the probability of finding such a subset by chance alone approaches 100%, rendering standard $p$-values of 0.05 meaningless.

Quantifying Selection Bias

The authors quantify this bias using order statistics. When a reporter selects the top $N_P$ results from a pool of $N_A$ datasets, the reported mean $\hat{\mu}_P$ is no longer an unbiased estimator of the algorithm's true performance. Instead, it is drawn from the extreme right tail of the distribution.

Bias of the reported mean $\hat{\mu}_P$ when selecting the top $N_P$ results from a pool of $N_A$ datasets.

As illustrated in the figure above, the selection process effectively samples from a region representing the top $N_P/N_A$ fraction of the distribution. This leads to a reported performance that is significantly inflated compared to the algorithm's actual average capability.

The Inspector Model

To address this, the paper proposes a "post-reporting" inspector model. Rather than trying to prevent bias at the time of submission, an "inspector" (such as a peer reviewer) validates the reported results against a new, independent set of datasets ($D_I$).

The magnitude of the reported mean $\hat{\mu}_P$ required to be considered statistically significant at level $\alpha$.

The inspector uses a two-sample test to determine if the gap between the reported results and the fresh results is suspiciously large. Figure (a) shows the standard significance thresholds that a reporter might target. The inspector model checks if the reported mean $\hat{\mu}_P$ is significantly higher than the mean $\hat{\mu}_I$ obtained from the fresh data. If the discrepancy exceeds a calculated threshold, the report is flagged as biased.

Experimental Validation

The authors validated their model using synthetic data and real-world benchmarks. In a case study comparing Logistic Regression (LR) and Gradient Boosting Trees (GBT) across 66 datasets, GBT was generally superior. However, by cherry-picking just 5 datasets, a biased reporter could produce a $p$-value of $\approx 2.6 \times 10^{-3}$, making LR appear significantly better. When the inspector model was applied with 5 fresh datasets, it successfully detected this selection bias in 95.9% of trials.

Conclusion

Cherry-picking undermines the scientific value of SOTA benchmarks. By shifting the focus from prevention to post-publication verification, the inspector model provides a robust statistical framework for maintaining research integrity. Future work will explore non-parametric extensions and applications in sequential testing environments like A/B testing.

Summary updated

Abstract

Statistical hypothesis testing serves as statistical evidence for scientific innovation. However, if the reported results are intentionally biased, hypothesis testing no longer controls the rate of false discovery. In particular, we study such selection bias in machine learning models where the reporter is motivated to promote an algorithmic innovation. When the number of possible configurations (e.g., datasets) is large, we show that the reporter can falsely report an innovation even if there is no improvement at all. We propose a `post-reporting' solution to this issue where the bias of the reported results is verified by another set of results. The theoretical findings are supported by experimental results with synthetic and real-world datasets.
BibTeX
@misc{komiyama2018simple,
  author = {Junpei Komiyama and Takanori Maehara},
  title = {{A Simple Way to Deal with Cherry-picking}},
  year = {2018},
  eprint = {1810.04996},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1810.04996},
}
Figure