Deep Ensembles on a Fixed Memory Budget: One Wide Network or Several Thinner Ones?

Nadezhda Chirkova, Ekaterina Lobacheva, Dmitry P. VetrovarXiv 2020 · arXiv

Abstract

This paper investigates the optimal allocation of a fixed parameter budget between a single wide neural network and an ensemble of thinner networks. The authors identify the Memory Split Advantage (MSA), demonstrating that ensembles of medium-sized networks consistently outperform individual large models across various tasks and architectures, including WideResNets and Transformers. The findings suggest that for a given memory constraint, ensembling is often a more efficient use of parameters than increasing model width.

Introduction

In modern deep learning, the prevailing wisdom is that "bigger is better." To improve performance, researchers typically either increase the size of a single model (e.g., making it wider) or train an ensemble of multiple models. However, real-world applications often operate under a fixed memory budget (total number of parameters).

The authors ask a fundamental structural question: Given a fixed number of parameters, what is the most effective way to spend them? Should one train one very wide neural network, an ensemble of several medium-width networks, or an ensemble of many very thin networks?

Methodology

The researchers investigated "memory splits"—different ways to divide a total parameter budget $B$ into $N$ networks, each of size $B/N$. They primarily scaled the width of the networks (number of filters or embedding dimensions) rather than depth, as depth scaling often introduces optimization difficulties.

To ensure the results were robust, they tested two scenarios:

  • Setting A (No tuning): Using the same hyperparameters for all split configurations to observe the "raw" effect.
  • Setting B (With tuning): Using grid search and Bayesian Optimization to find the best possible hyperparameters for each specific model size.

Performance was evaluated using standard metrics like Accuracy for classification and BLEU for translation, alongside Calibrated Negative Log-Likelihood (NLL) to assess uncertainty estimation quality.

Test BLEU vs. Memory Budget for Transformers. Ensembles of medium-sized networks (orange) consistently outperform single wide networks (blue) at higher budgets.

Key Insights

  • The MSA Effect is Real: For almost all standard budget sizes, splitting the budget into an ensemble outperforms a single large model. For example, a standard WideResNet-28-10 is less accurate than an ensemble of 16 thinner WideResNets with the same total parameters.
  • Individual Quality Saturates: As a single model grows wider, accuracy gains eventually plateau. The gains from ensembling typically "kick in" before the single model reaches its peak, allowing the ensemble to overtake individual performance.
  • Optimal Split Grows with Budget: There is no single "perfect" number of models. As the total parameter budget increases, the optimal number of models in the ensemble ($N$) also tends to increase.
  • Uncertainty Benefits: Ensembles on a fixed budget provide significantly better-calibrated uncertainty estimates (lower NLL) than a single wide network.
Test Accuracy vs. Ensemble Size for VGG on CIFAR-10 with tuned hyperparameters. The 'hump' indicates that for a fixed memory budget, an ensemble of several thinner networks is superior to a single wide one.

Key Results

  • WideResNet on CIFAR-100: A single standard WideResNet-28-10 (36.8M parameters) achieved 80.6% accuracy. An ensemble of 16 thinner models with the same total parameters achieved 82.52%.
  • Transformer on IWSLT'14: Ensembles of medium-width Transformers consistently outperformed both single wide Transformers and ensembles of many tiny Transformers, as shown by the performance "hump" in Figure 1.
  • Robustness: The MSA effect held true even when using advanced hyperparameter tuning like Bayesian Optimization, proving it is a structural advantage rather than an optimization fluke.

Limitations and Future Work

While the memory (parameters) is fixed, training $N$ separate models can be more computationally expensive in terms of wall-clock time and energy if not parallelized efficiently. Additionally, finding the optimal ensemble size $N$ currently requires empirical plotting. The authors suggest that developing a method to predict these optimal values without exhaustive training is a vital direction for future research. Finally, while this study focused on width, investigating how depth-based memory splits interact with the MSA effect remains an open question.

Summary updated

Abstract

One of the generally accepted views of modern deep learning is that increasing the number of parameters usually leads to better quality. The two easiest ways to increase the number of parameters is to increase the size of the network, e.g. width, or to train a deep ensemble; both approaches improve the performance in practice. In this work, we consider a fixed memory budget setting, and investigate, what is more effective: to train a single wide network, or to perform a memory split -- to train an ensemble of several thinner networks, with the same total number of parameters? We find that, for large enough budgets, the number of networks in the ensemble, corresponding to the optimal memory split, is usually larger than one. Interestingly, this effect holds for the commonly used sizes of the standard architectures. For example, one WideResNet-28-10 achieves significantly worse test accuracy on CIFAR-100 than an ensemble of sixteen thinner WideResNets: 80.6% and 82.52% correspondingly. We call the described effect the Memory Split Advantage and show that it holds for a variety of datasets and model architectures.
BibTeX
@misc{chirkova2020deep,
  author = {Nadezhda Chirkova and Ekaterina Lobacheva and Dmitry P. Vetrov},
  title = {{Deep Ensembles on a Fixed Memory Budget: One Wide Network or Several Thinner Ones?}},
  year = {2020},
  eprint = {2005.07292},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2005.07292},
}
Figure