Ensembling is one of the most reliable ways to boost machine learning performance. As demonstrated by Wang et al. in Wisdom of Committees: An Overlooked Approach To Faster and More Accurate Models, combining simple off-the-shelf models into committees or cascades can drastically outperform massive single architectures while cutting computational costs. However, a fundamental bottleneck arises when we have a large library of candidate models: how do we choose the optimal subset to form the ensemble?
If we have a pool of $N$ models, there are $2^N$ possible subensembles. Evaluating every single combination to find the absolute best team is computationally impossible for large $N$. Early work by Nayak et al. in Automated Model Selection Using Context-Dependent Behaviors established that finding the simplest adequate model configuration is an NP-hard search problem. This combinatorial explosion is a recurring challenge across domains, from traditional regression bagging (Hernández-Lobato et al. in Pruning in Ordered Regression Bagging Ensembles) to modern deep learning pipelines (Zhou et al. in The NiuTrans Machine Translation Systems for WMT21).
To bypass brute-force search, researchers leverage specific mathematical properties of model combinations to find near-optimal subsets efficiently.
The Geometry of Subsets: Submodularity and Greedy Selection
To bypass the $2^N$ search space, we can exploit the mathematical structure of the selection problem. One of the most powerful properties available is submodularity, which formalizes the intuitive concept of "diminishing returns."
Imagine you have a small ensemble of two models. Adding a third, diverse model might significantly expand the ensemble's coverage and correct its errors. However, if you already have a massive ensemble of fifty models, adding that same third model provides much less incremental value. When an objective function (like ensemble accuracy or task coverage) is submodular, a simple greedy algorithm—which iteratively adds the single best remaining model—is mathematically guaranteed to find a solution that is close to the theoretical optimum.
This principle is elegantly leveraged by Bachu et al. in Building a Winning Team: Selecting Source Model Ensembles using a Submodular Transferability Estimation Approach. By framing ensemble selection as a submodular optimization problem based on task alignment and model agreement, they provide an efficient greedy framework that bypasses exhaustive enumeration while maintaining high performance. Similarly, Lan et al. in Causal Q-Aggregation for CATE Model Selection demonstrate that greedy aggregation strategies can efficiently optimize weights for large model libraries even in complex causal inference scenarios.
Pruning and Diversity Ordering
Instead of building an ensemble from scratch, another approach starts with a large pool and prunes it down. The key to successful pruning is ensuring that the remaining models are highly diverse—meaning they fail in complementary ways.
As Tang et al. note in An analysis of diversity measures, maximizing diversity is theoretically tied to maximizing the ensemble's margin, but doing so directly is an NP-complete problem. To make this tractable, researchers use ordering heuristics. In Pruning in ordered bagging ensembles, the authors introduce "Orientation Ordering," a technique that sequences classifiers so that the most complementary models are placed first, allowing users to safely discard up to 85% of the pool while maintaining accuracy.
In modern deep learning, Liu et al. advance this concept in Hierarchical Pruning of Deep Ensembles with Focal Diversity. They use a "focal diversity" metric to measure how precisely models compensate for each other's specific failures. By applying an iterative hierarchical pruning strategy, they bypass exponential complexity, slashing computational overhead by over 50% while maintaining or even improving accuracy.
Meta-Heuristics and Quality-Diversity Optimization
When the relationship between models is highly non-linear, greedy heuristics can sometimes get trapped in local optima or overfit the validation data. To navigate these rugged search spaces, researchers turn to population-based optimization.
For instance, Purucker et al. introduce stochastic search methods in Q(D)O-ES: Population-based Quality (Diversity) Optimisation for Post Hoc Ensemble Selection in AutoML, using "quality diversity" algorithms to discover synergistic model groupings that incremental strategies miss. Similarly, Qasem et al. utilize nature-inspired optimization in Heterogeneous Ensemble Pruning based on Bee Algorithm for Mammogram Classification to find optimal subsets of heterogeneous classifiers. For complex architectures, Moudrík demonstrates in Evolving Non-linear Stacking Ensembles for Prediction of Go Player Attributes that genetic algorithms can successfully co-optimize both the selection of base learners and their non-linear aggregation functions.
Modern Frontiers: From LLM Stitching to Policy Committees
The challenge of combinatorial selection has taken on new urgency with the rise of foundation models and reinforcement learning. Rather than training massive monolithic systems, modern frameworks orchestrate specialized sub-components.
In the realm of Large Language Models, Su et al. introduce GPTailor: Large Language Model Pruning Through Layer Cutting and Stitching. Instead of standard weight pruning, the authors frame pruning as a zero-order optimization problem over a high-dimensional search space, cutting and stitching layers from multiple task-specific models to form a smaller, high-performing hybrid. For reasoning tasks, Wang et al. develop SLM-MUX: Orchestrating Small Language Models for Reasoning, utilizing a confidence-based selection mechanism to dynamically route tasks to the most effective subset of small language models, outperforming monolithic giants.
Finally, this paradigm extends to sequential decision-making. In Learning Policy Committees for Effective Personalization in MDPs with Diverse Tasks, Ge et al. address the selection bottleneck in multi-task reinforcement learning by training a committee of specialized policies. By clustering tasks in a latent space, they ensure optimal coverage, allowing a fast selection phase to identify the best expert policy for any given task during few-shot adaptation.
Key Foundational Work
Active Learning with Ensembles for Image Classification (2003) — contributed 1 concept elements to this direction.
Automated Model Selection Using Context-Dependent Behaviors (1992) — contributed 1 concept elements to this direction.