Abstract
This paper investigates the optimal allocation of a fixed parameter budget between a single wide neural network and an ensemble of thinner networks. The authors identify the Memory Split Advantage (MSA), demonstrating that ensembles of medium-sized networks consistently outperform individual large models across various tasks and architectures, including WideResNets and Transformers. The findings suggest that for a given memory constraint, ensembling is often a more efficient use of parameters than increasing model width.
Introduction
In modern deep learning, the prevailing wisdom is that "bigger is better." To improve performance, researchers typically either increase the size of a single model (e.g., making it wider) or train an ensemble of multiple models. However, real-world applications often operate under a fixed memory budget (total number of parameters).
The authors ask a fundamental structural question: Given a fixed number of parameters, what is the most effective way to spend them? Should one train one very wide neural network, an ensemble of several medium-width networks, or an ensemble of many very thin networks?
Methodology
The researchers investigated "memory splits"—different ways to divide a total parameter budget $B$ into $N$ networks, each of size $B/N$. They primarily scaled the width of the networks (number of filters or embedding dimensions) rather than depth, as depth scaling often introduces optimization difficulties.
To ensure the results were robust, they tested two scenarios:
- Setting A (No tuning): Using the same hyperparameters for all split configurations to observe the "raw" effect.
- Setting B (With tuning): Using grid search and Bayesian Optimization to find the best possible hyperparameters for each specific model size.
Performance was evaluated using standard metrics like Accuracy for classification and BLEU for translation, alongside Calibrated Negative Log-Likelihood (NLL) to assess uncertainty estimation quality.
Key Insights
- The MSA Effect is Real: For almost all standard budget sizes, splitting the budget into an ensemble outperforms a single large model. For example, a standard WideResNet-28-10 is less accurate than an ensemble of 16 thinner WideResNets with the same total parameters.
- Individual Quality Saturates: As a single model grows wider, accuracy gains eventually plateau. The gains from ensembling typically "kick in" before the single model reaches its peak, allowing the ensemble to overtake individual performance.
- Optimal Split Grows with Budget: There is no single "perfect" number of models. As the total parameter budget increases, the optimal number of models in the ensemble ($N$) also tends to increase.
- Uncertainty Benefits: Ensembles on a fixed budget provide significantly better-calibrated uncertainty estimates (lower NLL) than a single wide network.
Key Results
- WideResNet on CIFAR-100: A single standard WideResNet-28-10 (36.8M parameters) achieved 80.6% accuracy. An ensemble of 16 thinner models with the same total parameters achieved 82.52%.
- Transformer on IWSLT'14: Ensembles of medium-width Transformers consistently outperformed both single wide Transformers and ensembles of many tiny Transformers, as shown by the performance "hump" in Figure 1.
- Robustness: The MSA effect held true even when using advanced hyperparameter tuning like Bayesian Optimization, proving it is a structural advantage rather than an optimization fluke.
Limitations and Future Work
While the memory (parameters) is fixed, training $N$ separate models can be more computationally expensive in terms of wall-clock time and energy if not parallelized efficiently. Additionally, finding the optimal ensemble size $N$ currently requires empirical plotting. The authors suggest that developing a method to predict these optimal values without exhaustive training is a vital direction for future research. Finally, while this study focused on width, investigating how depth-based memory splits interact with the MSA effect remains an open question.
Summary updated