A Boo(n) for Evaluating Architecture Performance

Ondrej Bajgar, Rudolf Kadlec, Jan KleindienstICML 2018

Deep learning research currently suffers from a "statistical lottery." Papers frequently report only the absolute best result from multiple training runs—a practice that is inherently noisy and disproportionately rewards researchers with the most computational resources. To address this, researchers have introduced $Boo_n$ (Expected Best-out-of-$n$), a robust metric based on order statistics. $Boo_n$ provides a fairer, more reproducible way to compare model architectures by estimating the performance a practitioner can realistically expect when picking the best of $n$ trained models.

The Problem: The Statistical Lottery and Compute Bias

Training a deep neural network is a stochastic process. Even with identical hyperparameters, factors like random weight initialization and data shuffling lead to significantly different final accuracies.

Performance variance across different runs

The current standard in the field is to run several experiments and report the "best single model" performance. This approach is scientifically flawed for three main reasons:

  1. The Compute Bias: The expected value of the "best" result artificially increases simply by running more experiments. A lab with 100 GPUs can "outperform" a lab with 10 GPUs just by finding a luckier outlier.
  2. Lack of Reproducibility: A single lucky run does not characterize the general behavior of an architecture. Another researcher trying to replicate the result is highly unlikely to hit that exact same outlier.
  3. The "SOTA" Margin Problem: The random variation between runs is often larger than the performance gains reported in many state-of-the-art (SOTA) papers. This makes it unclear if a newly proposed architecture is actually superior or just "luckier."

The Solution: Introducing $Boo_n$

To combat these issues, the authors propose $Boo_n$ (normalized expected best-out-of-$n$ performance). Instead of reporting the maximum value found across an arbitrary and often unreported number of runs, $Boo_n$ estimates what the expected maximum would be if you specifically trained exactly $n$ models.

Key technical aspects of this approach include:

  • Order Statistics: The method leverages the mathematical theory of order statistics to calculate the expected maximum of a performance distribution.
  • Non-parametric Estimator: The authors provide a formula to estimate $Boo_n$ from $m$ experimental runs (where $m \ge n$). It essentially acts as a weighted average of all test results, where the weights are determined by the probability of each result being the maximum in a sample of size $n$.
  • Validation-Test Handling: Crucially, the metric accounts for the real-world practice of picking the best model based on validation performance and then reporting its test performance.
  • Parametric Alternative: For cases where the performance distribution is assumed to be Gaussian, they provide a simplified version where $Boo_n$ becomes a function of the mean, standard deviation, and a constant.

Experimental Validation and Key Insights

The authors validated their approach using two distinct domains to demonstrate the prevalence of performance noise: image recognition using ResNet on CIFAR-100, and reading comprehension using the Attention Sum (AS) Reader on the Children’s Book Test (CBT) - Common Nouns.

Removing the "Race for Compute"

Reporting the best-of-$m$ results incentivizes a brute-force approach to research. $Boo_n$ removes this incentive entirely. Running more experiments ($m$) simply makes the estimate of $Boo_n$ more precise, rather than artificially inflating the score.

While the mean is a standard statistical tool, researchers often avoid it because it doesn't reflect production reality—in practice, we do train several models and deploy the best one. $Boo_n$ captures this "best-of" intuition while remaining statistically sound.

Quantifying the Flaw and Proving Robustness

Experiments revealed that the interquartile range of accuracies for a fixed model was around 1% absolute. Given that many papers claim "superiority" with margins of 0.5% or less, much of the reported progress in the field may simply be noise.

Stability of Boo_n vs. Best Single Model

The authors showed that for the AS Reader, the expected "best" result increased from 61.3% (1 run) to 63.5% (20 runs). This 2.2% "gain" is purely due to the number of experiments, not architectural improvement. In contrast, $Boo_n$ (specifically $Boo_5$) remained remarkably stable regardless of whether the researchers ran 10 or 100 experiments, whereas the "best single model" metric kept climbing.

Furthermore, the correlation between validation and test performance was sometimes surprisingly low (as low as 0.10). This highlights why picking the "best" model on validation doesn't always lead to the best test result—a nuance that $Boo_n$ handles correctly.

Limitations and Future Directions

While $Boo_n$ is a significant step forward, it has limitations:

  • Hyperparameter Tuning: It works well for random hyperparameter search but doesn't fully account for more complex, non-independent tuning strategies like Bayesian optimization.
  • Computational Cost: To calculate a reliable $Boo_n$, you still need to run multiple experiments ($m > n$). For extremely large models where only a single training run is feasible, this metric cannot be applied.
  • Choice of $n$: The value of $n$ is somewhat arbitrary. The authors suggest $n=5$ as a reasonable standard for "overnight" replication, but different sub-fields might require different standards.

Ultimately, the authors suggest that future research should focus on better characterizing the entire performance distribution of architectures, rather than searching for single-point estimates that can be easily manipulated by compute budgets.

Summary updated

BibTeX
@inproceedings{bajgar2018boo,
  author = {Ondrej Bajgar and Rudolf Kadlec and Jan Kleindienst},
  title = {{A Boo(n) for Evaluating Architecture Performance}},
  booktitle = {International Conference on Machine Learning},
  year = {2018},
}
Figure