Closing the Ensemble Optimality Gap

15 concepts5 papers1 proposal

When building complex machine learning systems, our natural instinct is to divide and conquer. To improve accuracy or handle complex tasks, we often train an ensemble of multiple models and average their predictions, or we construct large neural networks by combining smaller, specialized modules.

Typically, we optimize each component in isolation—finding the best hyperparameters for each model or training each module on a specific sub-task—and then stick them together. We assume that a collection of individually perfect parts will naturally yield an optimal whole.

However, this assumption is fundamentally flawed. In practice, a severe individual-centric optimization bias misaligns the training of components with the ultimate goal of the system. When components are optimized independently, they often learn redundant features, replicate each other's errors, and fail to cooperate. To build truly effective compound systems, we must bridge what Laurits Fredsgaard et al. term the ensemble optimality gap: the discrepancy between what is optimal for an individual model and what is optimal for the collective system.


The Core Problem: Redundancy vs. Cooperation

To understand why independent optimization fails, imagine assembling a basketball team by independently picking the five highest-scoring point guards in the league. While individually brilliant, they will perform poorly as a team because they share the same strengths and weaknesses; none of them are optimized to rebound, block shots, or pass to one another.

Traditional machine learning pipelines make this exact mistake. Standard hyperparameter optimization (HPO) searches for settings that minimize the error of a single model in isolation. If you use this strategy to build an ensemble, you end up with a team of identical point guards.

As Christian Gagné et al. demonstrate, the best "team players" in an ensemble are often configurations that would be discarded by traditional HPO because their standalone performance is mediocre. When models are forced to optimize for the collective objective, the search process prioritizes diversity and complementary performance over individual peak accuracy. The models naturally specialize, learning to cover the specific blind spots of their peers.


Joint Hyperparameter Optimization

How do we force models to optimize for the team rather than themselves? One powerful approach is to change the objective function during the hyperparameter search or tuning phase.

Instead of evaluating a model configuration on its standalone validation error, the search objective can be explicitly tied to the ensemble's performance. Christian Gagné et al. proposed a sequential, round-robin Bayesian optimization framework where the objective is defined as the validation error of the entire ensemble when a specific member is replaced.

$$\text{Objective} = \mathcal{L}\Big(\text{Ensemble} \setminus \{m_{\text{old}}\} \cup \{m_{\text{new}}\}\Big)$$

By optimizing this collective objective, the search algorithm actively hunts for hyperparameters that produce models specializing in exactly the data points where the current ensemble is weak.

Global Regularization and Calibration

The need for joint tuning extends beyond architecture search to foundational training choices like weight decay, early stopping, and temperature scaling. Laurits Fredsgaard et al. showed that when these hyperparameters are tuned independently for each member of a deep ensemble, the resulting choices fail to fully leverage the ensemble's potential for uncertainty quantification.

By jointly tuning these parameters for the entire ensemble collectively, the authors close the optimality gap, yielding significantly better calibration and generalization. Joint regularization ensures that individual models do not overfit to the same patterns, preserving the diversity essential for reliable ensemble predictions.


Global Post-Refinement

The ensemble optimality gap doesn't just exist at the hyperparameter level; it also plagues the internal weights of combined systems. This occurs in two main paradigms: sequential ensembles (like boosting) and modular neural networks.

Unfreezing Greedy Ensembles

Standard boosting algorithms are notoriously greedy. They train a classifier, freeze its weights, and then train the next classifier to correct the remaining residual errors. Rong Zhang et al. point out that this one-way optimization treats the ensemble as a collection of static parts rather than a single differentiable system.

To fix this, the authors introduce a post-optimization phase. After the sequential training concludes, they unfreeze the internal parameters of all constituent classifiers and their combination weights, using gradient descent to fine-tune them simultaneously. This global adjustment allows early classifiers to retroactively adapt to the learning outcomes of subsequent models, creating a highly integrated system where a handful of finely-tuned models can outperform much larger, traditionally boosted sets.

Greedy Sequential Training (Standard):
[Model 1 (Frozen)] -> [Model 2 (Frozen)] -> [Model 3 (Frozen)]

Global Post-Refinement (Joint):
[Model 1 (Tuning)] <---> [Model 2 (Tuning)] <---> [Model 3 (Tuning)]
                       \          |          /
                        \---> [Joint Loss] <--/

Polishing Modular Networks

A parallel problem occurs when building large neural networks out of specialized functional blocks. In his seminal work on scaling walls in speech recognition, Alex Waibel demonstrated that while modular construction provides a powerful "divide and conquer" strategy for training, simply snapping pre-trained modules together using integration units is insufficient.

To make the modules cooperate, the entire network must undergo a final global optimization phase where all weights are unfrozen. This holistic training pass polishes the interactions between disparate functional blocks, reconciling local features with global task requirements.

This principle remains highly effective for modern incremental architectures. For instance, when constructing deep networks for complex character recognition, Saikat Roy et al. utilize a final training phase on the fully assembled network using a very low learning rate. This step refines the weights across all layers simultaneously, ensuring that the boundaries between incrementally added layers are smoothed out and optimized for the final global task.


Why It Works: A Unified Perspective

Whether you are tuning the learning rate of a deep neural network or selecting the regularization strength of an ensemble, the underlying physics of the problem are identical:

Approach Individual-Centric (Sub-optimal) Collective/Joint (Optimal)
Hyperparameters Maximize standalone accuracy; leads to redundant, highly correlated models. Optimize contribution to the group; forces diversity and error-correction.
Model Weights Freeze early components greedily; forces later components to do all the heavy lifting. Unfreeze and fine-tune globally; allows components to co-adapt and share the objective.

Joint optimization works because it acknowledges that the combination operator is part of the system. When you optimize components in isolation, you ignore how they interact through the final voting or layer-aggregation mechanism. By shifting the optimization target to the final collective output, you allow gradient descent or black-box optimizers to exploit the synergies between parts, effectively closing the ensemble optimality gap.

Key Foundational Work

Optimizing boosting with discriminative criteria (2004) — contributed 3 concept elements to this direction.
Consonant Recognition by Modular Construction of Large Phonemic Time-Delay Neural Networks (1988) — contributed 2 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_ensemble_optimality_gap_f84a5a32,
  title = {ensemble optimality gap},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/ensemble-optimality-gap/rdr_f84a5a328a0d45b8820d98df13752774},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_ensemble_optimality_gap_f84a5a32_v1,
  title = {ensemble optimality gap},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/ensemble-optimality-gap/rdr_f84a5a328a0d45b8820d98df13752774/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake