Efficient Inference for Model Ensembles

This direction is merged into another direction: ensemble inference overhead. Its last version stays readable and citable here.

Version 1 of 1577 papers590 conceptsfrozen 2026-10-04

In machine learning, there is a recurring "ensemble tax." We know that combining multiple models—whether by averaging their predictions, voting, or layering them—leads to better generalization, higher robustness, and fewer hallucinations. However, this "wisdom of the crowd" usually comes with a linear increase in cost: running ten models takes ten times the compute, ten times the memory, and ten times the latency.

Recent research has focused on breaking this linear relationship. The goal is to capture the performance of an ensemble while maintaining the efficiency of a single model. This is achieved through three primary mechanisms: merging models in parameter space, compressing collective knowledge into a single student, and intelligently routing queries to only the necessary experts.

Merging Models in Parameter Space

The most direct way to avoid the ensemble tax is to combine models before they ever reach inference. If we have two neural networks that have learned different useful features, we might try to average their weights.

Pavel Izmailov and colleagues introduced Stochastic Weight Averaging (SWA) in Averaging Weights Leads to Wider Optima and Better Generalization to exploit the geometry of the loss landscape. They observed that standard training often ends up near the edge of a "flat" region of low loss. By averaging weights along the trajectory of the optimization, SWA finds a point in the center of this region. This "wider" optimum generalizes better to new data, providing the benefits of an ensemble within a single set of weights.

Building on this, Alexandre Ramé proposed Diverse Weight Averaging (DiWA) in Diverse Weight Averaging for Out-of-Distribution Generalization. While SWA averages weights from a single training run, DiWA averages weights from independent runs. This captures functional diversity—the different "perspectives" models have on the data—without requiring multiple forward passes at test time.

Compressing the Crowd via Amalgamation

When models cannot be directly averaged—perhaps because they have different architectures—researchers turn to Knowledge Amalgamation. The idea is to treat the ensemble as a "teacher" and train a single "student" model to mimic its collective output.

In MEAL: Multi-Model Ensemble via Adversarial Learning, Xiangyang Xue and his team used adversarial learning to align the intermediate features of a student network with those of a diverse ensemble. By forcing the student to learn the inter-class relationships captured by the ensemble's "soft labels," they achieved ensemble-level accuracy with a single-model footprint. Similarly, Guocong Song demonstrated in Collaborative Learning for Deep Neural Networks that multiple "heads" of a single network can train together to reach a consensus, effectively ensembling during training to produce a more robust single model for deployment.

Dynamic Routing and Selection

Not every query requires the full power of a massive ensemble. A simple question like "What is 2+2?" doesn't need a committee of GPT-4-level models. Dynamic Routing attempts to slash costs by selecting the smallest or most appropriate model for a specific input.

Jinwu Hu introduced Dynamic Ensemble Reasoning (DER) in Dynamic Ensemble Reasoning for LLM Experts, which treats ensembling as a sequential decision process. Instead of running all experts in parallel, a reinforcement learning agent routes the query through a chain of specialized models, stopping as soon as a reliable answer is found. This approach can reduce computational parameters by up to 7-fold.

Similarly, Shuhao Chen developed RouterDC in RouterDC: Query-Based Router by Dual Contrastive Learning for Assembling Large Language Models. By using contrastive learning to map queries to the models most likely to answer them correctly, RouterDC avoids the "redundant inference" that plagues traditional majority-voting ensembles. For commercial applications, Lingjiao Chen proposed FrugalML in FrugalML: How to Use ML Prediction APIs More Accurately and Cheaply, which learns to route data between different paid APIs to maximize accuracy while staying strictly under a financial budget.

Efficient Self-Consistency

Even when using a single model, we often "ensemble" it with itself by sampling multiple reasoning paths—a technique called self-consistency. However, generating dozens of paths is slow and expensive.

Amir Taubenfeld addressed this in Confidence Improves Self-Consistency in LLMs by introducing Confidence-Informed Self-Consistency (CISC). By weighting the model's votes based on its internal confidence scores, CISC can reach the same level of accuracy as standard voting while requiring 40% fewer samples. This demonstrates that even within the "self-ensemble" paradigm, better aggregation logic can directly translate to lower inference costs.

Why It Works: Diversity and Redundancy

The success of these methods rests on two pillars: Diversity and Redundancy. Ensembles work because different models fail in different ways; as Bo Xiong shows in Predictive Multiplicity of Knowledge Graph Embeddings in Link Prediction, even models with identical accuracy can have "hidden disagreements."

The research direction described here recognizes that while diversity is essential for performance, the redundancy of running multiple full models is an engineering bottleneck. By moving the "merging" step earlier in the pipeline—either into the weights, the training process, or a smart routing layer—researchers are making the high-performance "wisdom of the crowd" practical for real-world, resource-constrained deployment.

Papers in this version

Showing 60 of 577 papers.

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_efficient_inference_for_model_ensembles_1ecceb3a,
  title = {Efficient Inference for Model Ensembles},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/efficient-inference-for-model-ensembles/rdr_1ecceb3a04c94dc99002c65fcfb84415},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_efficient_inference_for_model_ensembles_1ecceb3a_v1,
  title = {Efficient Inference for Model Ensembles},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/efficient-inference-for-model-ensembles/rdr_1ecceb3a04c94dc99002c65fcfb84415/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}